Can't start mysql server

Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock

If you get the above error on Ubuntu and you have tried all the things suggested by others (permissions, recreating the socket file, reboot, bind address change) you may want to try stopping apparmor and starting again mysql.

service apparmor stop
service mysql start

later edit
to do: instructions for correct configuration of apparmor, since stopping the service is a bit barbarian(but efficient).

mEgo is sky high

Yes ladies and gents, it is true! mEgo Inc, my former company where I have worked for the last 2 years as sysadmin, made the move to Amazon Elastic Compute Cloud (aka EC2) today!!!11oneone It's a moment of great joy to see this thing happening now at the beginning of 2010.

Best wishes to the mEgo team!

LE: Proof
PING mego.com (67.202.39.53) 56(84) bytes of data.
64 bytes from ec2-67-202-39-53.compute-1.amazonaws.com (67.202.39.53): icmp_seq=1 ttl=48 time=10.6 ms


WordPress and dubious themes

If you are like me and deal with a lot of WordPress themes chances are that you have encountered several themes that contain something like:
eval(base64_decode('ZnVuY3Rpb24gY2hlY2tfdGhlbWVfaGVhZGV
yKCkgeyBpZiAoIShmdW5jdGlvbl9leGlzdHMoImZ1bmN0aW9uc19maWxlX2V4
aXN0cyIpICYmIGZ1bmN0aW9uX2V4aXN0cygidGhlbWVfZm9vdGVyX3QiKSkpI
HsgdGhlbWVfdXNhZ2VfbWVzc2FnZSgpOyBkaWU7IH0gfQ=='))
Now, all the people, or at least most of them would advice not to use such things on your WordPress and probably they are right. But if you are a curios person like me you have to know what that long string means.
function check_theme_header() { if (!(function_exists("functions_file_exists") && function_exists("theme_footer_t"))) { theme_usage_message(); die; } }
Meh, not a big deal. Someone is very obsessed about his links in the footer of your WordPress theme. He want's to make sure you keep his precious links in place. Whatever!
I find this very rude however and a headache for anyone that installs wordpress themes, especially free ones. A simple "Please keep the footer intact" would do better. People are going to replace the footer anyway if that's what they desire. No one wants some strange text that he doesn't know what's doing on his server.

And because I like to have at least one useful thing in my posts here is a link to an online decoder, the same I have used to decode the example: http://www.motobit.com/util/base64-decoder-encoder.asp

To quickly find such precious bits of information you can run something like:
grep "eval(base64_decode" -r /path/to/some/dir 2>/dev/null

LE: An so I discover another bug in my blogger theme ...

ISPConfig3 running on Ubuntu 9.10 on AWS

This is an idea on how to set ISPConfig 3 on Amazon EC2 for web hosting. It's still work in progress and I await for any constructive feedback. Since English is not my first language there might be some grammatical errors.

Prerequisites:

• AWS Account
• Some Linux experience

Part I - Setting up AWS

I have done all the setup of instance and ebs from the Amazon Management Console. It has a very intuitive interface and it's so easy to work with.

Optional: You can purchase reservation for an instance if you are planing to run it 24/7. To do so go to Instances -> Reserved Instances and click on Purchase Reserved Instance. For example a c1.medium instance reservation for 1 year costs 455$ one time fee and the hourly cost of the instance goes to 0.06$/hour. It's your decision, do the math if it's worth it or not to go with reserved instances.

1. From the Management Console go to Volumes and Create Volume. I created a 40 GB volume in us-east-1a zone, choose an appropriate value for your needs.
2. Next step which is also optional is to get an Elastic IP for your instance. Go to Elastic IPs and click Allocate New Address.
3. You will need a keypair to access your instance. You can create one before launching the instance or create one when you also setup the new instance. Your choice. I created one before by going to Key Pairs -> Create Key Pair.

Part II - Launching the instance

Instance type used was c1.medium, the AMI for it was Ubuntu 9.10 Karmic 32bit ami-1515f67c from Canonical Images for Amazon EC2.
Go to Instances and press Launch Instance and make sure you set it to use ami-1515f67c. If you need to use a different zone or a 64 bit instance you can check the alestic page which is a wonderful resource for Ubuntu images for EC2. I would also suggest to join their EC2 Ubuntu Google Grup if you are serious about AWS + Ubuntu.
After the instance is up and running attach the Elastic IP and the Volume you have created to it. To attach the IP go to Elastic IPs and select Associate, for the EBS go to Volumes. I attached the EBS as /dev/sdb to the instance.

Later Edit: You will need to also modify the Security Groups and permit access on ports 22(SSH), 80(HTTP), 8080(ISPConfig 3), 20-21(FTP) -> still need more work to support PASV.

Part III - Install ISPConfig 3

Before proceeding with ISPConfig 3 setup connect to your instance and update the packages.

sudo su
apt-get update
apt-get upgrade

Note: You will have to connect to your instance using the ubuntu username and the key pair you generated for that instance. It won't allow you to connect directly as root.

To install the ISPConfig 3 I followed the HowtoForge excellent tutorial with a small exception: I skipped the part of setting up the quota since I was planning to use the EBS as storage for sites and MySQL databases. The rest of the tutorial was just perfect.

Part IV - Setting up the EBS and ISPConfig to work with it

Most of the stuff and talk about EBS and MySQL snapshots is discussed on Amazon developer forums in the thread called Tutorial: Running MySQL on Amazon EC2 with EBS (Elastic Block Store). Make sure you read the discussion before or after you are done with this post.

apt-get install dmsetup
modprobe dm_mod
modprobe dm_mirror
modprobe dm_snapshot
mkdir /dev/.static/dev/mapper -pv

Create ext3 filesystem:
mkfs.ext3 /dev/sdb

Setup it with dmsetup
echo 0 `blockdev --getsize /dev/sdb` linear /dev/sdb 0 | dmsetup create ebs

Add the records to fstab
echo "/dev/mapper/ebs /ebs ext3 noatime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 0 0" >> /etc/fstab

Setup quota
touch /ebs/aquota.user /ebs/aquota.group
chmod 600 /ebs/aquota.*
mount -o remount /ebs

Enable quota on ebs
quotacheck -avugm -F vfsv0 /ebs
quotaon -avug /ebs

For websites:
mkdir /ebs/sites
ln -s /ebs/sites /var/www/clients

Jailkit:
Set Jailkit chroot home: /ebs/home/[username]
mkdir /ebs/home

MySQL:
/etc/init.d/mysql stop
mkdir /ebs/mysql/data /ebs/mysql/log -pv
mv /var/lib/mysql /ebs/mysql/data
ln -s /ebs/mysql/data/mysql /var/lib/mysql
mv /var/log/mysql /ebs/mysql/log
ln -s /ebs/mysql/log/mysql /var/log/mysql


To make a snapshot of the database, go to the mysql console (msqyl -p)
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;
SYSTEM dmsetup suspend /dev/mapper/ebs

Take your snapshot of the EBS

SYSTEM dmsetup resume /dev/mapper/ebs
UNLOCK TABLES;

Note:This is still work in progress, any advices on how to improve it are greatly appreciated and any constructive criticism also.

ISPConfig3: Updating to myDNS-ng

If you have followed the tutorials "the perfect setup" for installing ISPConfig 3 chances are that you have an older myDNS version (1.1.0 or so). Since I had a few problems with that version I decided to update to myDNS-ng, the latest version.

In this article I will refer to rpmbuilder user, a user I create to compile and/or make RPMs. For more information on how to set it up check this link.

Also the instructions are for CentOs 5.4(but will probably work for any other 5.x version).

su - rpmbuilder
cd /tmp
wget http://downloads.sourceforge.net/project/mydns-ng/mydns/1.2.8.27/mydns-1.2.8.27.tar.gz?use_mirror=garr
tar zxvf mydns-1.2.8.27.tar.gz
cd mydns-1.2.8
./configure
make

Everything should be ok untill now. I had no problems to compile it. Then I decided to use checkinstall.
sudo yum install checkinstall
checkinstall
It will ask you a few questions, answer them and you should have a rpm package in /usr/src/redhat/RPMS. It will tell you anyway where he created the rpm.
After that I removed the old mydns packages and installed the new one as root:
service mydns stop
yum remove mydns mydns-mysql
yum install /usr/src/redhat/RPMS/i386/mydns-1.2.8-1.i386.rpm --nogpgcheck
After that I needed to create mydns start script:
vi /etc/init.d/mydns
chmod +x /etc/init.d/mydns
I used the script posted by rukus77 on howtoforge. Make sure you read all the comments on that forum (especially the one containing [ -f /usr/sbin/mydns ] || exit 1).
The last step is to run ISPConfig 3 setup:
cd ispconfig3_install/
cd install
php -q update.php
When asked if to reconfigure services answer yes, the rest you can answer no.

That would be all, thanks for reading.

Install Postfix, Courier-IMAP on CentOS 5

Platform: CentOS 5.4 i386
Extra repositories used: utterramblings and rpmforge
You will end up with Postfix 2.6.5, Courier IMAP 4.6.0, authlib 0.62.4 and maildrop 2.2.0

First of all you will need to install gcc++:
yum install gcc-c++ redhat-rpm-config
Create a user to compile packages if you don't have one already. I used rpmbuilder for the account name.
useradd -m -s /bin/bash rpmbuilder
passwd rpmbuilder
It will need some privileges so type visudo and add this line somewhere (I added it after root):
rpmbuilder ALL=(ALL) ALL
Create the directory structure:
su - rpmbuilder
mkdir $HOME/rpm
mkdir $HOME/rpm/SOURCES
mkdir $HOME/rpm/SPECS
mkdir $HOME/rpm/BUILD
mkdir $HOME/rpm/SRPMS
mkdir $HOME/rpm/RPMS
mkdir $HOME/rpm/RPMS/i386
echo "%_topdir $HOME/rpm" >> $HOME/.rpmmacros
Start downloading the needed courier packages(under the user rpmbuilder):
cd /tmp
wget http://sourceforge.net/projects/courier/files/maildrop/2.2.0/maildrop-2.2.0.tar.bz2/download
wget http://sourceforge.net/projects/courier/files/imap/4.6.0/courier-imap-4.6.0.tar.bz2/download
wget http://sourceforge.net/projects/courier/files/authlib/0.62.4/courier-authlib-0.62.4.tar.bz2/download
These were the latest versions when I wrote the article, feel free to browse sourceforge and get what version you want.
Start building and installing the RPMs:
sudo rpmbuild -ta courier-authlib-0.62.4.tar.bz2
cd $HOME/rpm/RPMS/i386
sudo rpm -ivh courier-authlib-0.62.4-1.i386.rpm courier-authlib-mysql-0.62.4-1.i386.rpm courier-authlib-devel-0.62.4-1.i386.rpm
cd /tmp
rpmbuild -ta courier-imap-4.6.0.tar.bz2
cd $HOME/rpm/RPMS/i386
sudo rpm -ivh courier-imap-4.6.0-1.i386.rpm
cd /tmp
sudo rpmbuild -ta maildrop-2.2.0.tar.bz2
cd $HOME/rpm/RPMS/i386
sudo rpm -ivh maildrop-2.2.0-1.i386.rpm

Done with Courier. Moving to postfix...

Download source RPM from this location:
cd /tmp
wget http://ftp.wl0.org/official/2.6/SRPMS/postfix-2.6.5-1.src.rpm
rpm -ivh postfix-2.6.5-1.src.rpm
I've made a few changes to the postfix.spec file (enabled mysql_redhat, vda patch) and also installed the required dependencies:
sudo yum install db4-devel mysql-shared mysql-devel
Building the postfix-2.6.5 rpm:
cd $HOME/rpm/SPECS
rpmbuild -ba postfix.spec
You should have the rpm in $HOME/rpm/BUILD so cd there and type:
sudo rpm -ivh postfix-2.6.5-1.rhel5.i386.rpm

In the end enable required services as root user:
chkconfig --levels 235 courier-authlib on
chkconfig --levels 235 sendmail off
chkconfig --levels 235 postfix on
chkconfig --levels 235 saslauthd on
chkconfig --levels 235 courier-imap on
service sendmail stop
service postfix start
service saslauthd start
service courier-authlib start
service courier-imap start

If everything went ok when telneting on port 25 of localhost you should get something similar to this:
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220 lightbringer.inno-solutions.ro ESMTP Postfix (2.6.5)
ehlo localhost
250-lightbringer.inno-solutions.ro
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

NOTE 1:
If you already have any of the packages install use rpm -Uvh instead of rpm -ivh.

Cloud providers compared

Cloud Providers

Amazon offers EC2(Elastic Compute Cloud) as IaaS(Infrastructure as a Service). Maybe the most well known cloud provider, provides a wide variety of instances to use starting from 10 cents/hour up to 80 cents/hour.
Rackspace offers Cloud Servers, the equivalent of Amazon's EC2. The prices start from 1.5 cents/hour for 256 MB instance and go up to 96 cents/hour for their top instances.
GoGrid have, in their customer opinion, best UI(User Interface). Their prices seem a bit high, but with their prepaid plans prices go down to 10 cents/hour, similar to Amazon.

All three providers offer file storage also:
-Amazon offers S3 (15 cents/GB)
-Rackspace has Cloud Files(15 cents/GB)
-GoGrid has Cloud Storage in Beta (15 cents/GB)
So not much to say here. Prices are the same for storing.

But for uploading / downloading the files you store they also charge a tax.
-S3 will cost 1 cent per 1000 PUT/POST/COPY/LIST requests and 1 cent for 10 000 GET requests(when a user requests an image). DELETE is free :)
-Rackspace will cost 2 cents per 1000 PUT/POST/LIST requests and it's FREE for GET requests ... which I believe is an advantage for mEgo.
-Couldn't find any reasonable info about GoGrid on this matter.

Another important aspect is the bandwidth cost:
-Amazon 10c/17c for each 1 GB uploaded/downloaded
-Rackspace 8c/22c for each 1 GB uploaded/downloaded
-GoGrid with no plan you get to pay FREE/50c for each GB. But for 200$ a month we can get down to 20c/GB for downloads(Transfer 1TB plan).

Case study: 2 web servers each with 2 GB of RAM and one database server with 8 GB of RAM, 500 GB file storage, 400GB inbound transfers and 1TB outbound transfer.

Amazon:
2x c1.medium = 288$ (2 x 0.4$/h x 720h)
1x c1.high = 576$ (well this one has 7GB of RAM) (0.8$/h x 720h)
500GB file storage = 75$
400GB inbound = 40$
1TB outbound = 170$
Total 1149$/month

Rackspace:
2x 2048MB = 172$(2 x 0.12$/h x 720)
1x 8192MB = 345$(0.48$/h x 720)
500 GB file storage = 75$
400GB inbound = 32$
1TB outbound = 220$
Total 844$/month

GoGrid - for this one I will make use of their prepaid plans Advanced Cloud(499$/month for 5 000 RAM hours) and Tranfer 1TB plan(200$/month for 1TB outbound)
(2 x 2GB + 1x 8GB) * 720 = 8640 RAM hours at a price of 0.10$/hour which means 864$
500GB file storage = 75$
400GB inbound = 0$
1TB outbound = 200$
Total 1139$/month
LE: See Michael's comment for additional info regarding GoGrid

More Apache benchmarks using EC2 instances

Back with more benchmarks ... This time I've tested several EC2 instances, using the AMIs provided by scalr for application roles(app and app64). Site used for testing was mEgo.
c1.medium results:

Concurrency Level: 200
Time taken for tests: 311.715596 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 306810000 bytes
HTML transferred: 303610000 bytes
Requests per second: 32.08 [#/sec] (mean)
Time per request: 6234.312 [ms] (mean)
Time per request: 31.172 [ms] (mean, across all concurrent requests)
Transfer rate: 961.19 [Kbytes/sec] received

m1.large results:

Concurrency Level: 200
Time taken for tests: 420.241673 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 308313369 bytes
HTML transferred: 305097689 bytes
Requests per second: 23.80 [#/sec] (mean)
Time per request: 8404.834 [ms] (mean)
Time per request: 42.024 [ms] (mean, across all concurrent requests)
Transfer rate: 716.46 [Kbytes/sec] received

c1.xlarge results:

Concurrency Level: 200
Time taken for tests: 70.404865 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 306810000 bytes
HTML transferred: 303610000 bytes
Requests per second: 142.04 [#/sec] (mean)
Time per request: 1408.097 [ms] (mean)
Time per request: 7.040 [ms] (mean, across all concurrent requests)
Transfer rate: 4255.66 [Kbytes/sec] received

m1.xlarge results:

Concurrency Level: 200
Time taken for tests: 215.153753 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 308098602 bytes
HTML transferred: 304885162 bytes
Requests per second: 46.48 [#/sec] (mean)
Time per request: 4303.075 [ms] (mean)
Time per request: 21.515 [ms] (mean, across all concurrent requests)
Transfer rate: 1398.43 [Kbytes/sec] received

So to sum things up:
c1.medium can serve about 32 requests per second for 0.20$ per hour.
m1.large can serve less than c1.medium, around 24 requests per second for 0.40$ per hour ... not nice :(
m1.xlarge can serve 47 requests per second, but it will cost you 0.80$ per hour.
The champion is c1.xlarge can serve 142 requests per second at same price as m1.xlarge, 0.80$ pe hour.

I didn't include m1.small benchmarks, but from a previous post I can tell you it only server 6 requests per second, not a worthy instance.

My advice would be to go with c1.medium instances since they offer best price/performance ratio. But do the math for yourself!

MySQL benchmarks using Amazon EC2 instances

Here are some tests I've run on Amazon using AMIs provided by scalr for the mysql role. I've used the benchmark scripts supplied by MySQL located in /usr/share/mysql/sql-bench. I had to install a package before running the tests:
apt-get install libdbd-pg-perl

After that everything was simple:
root@ec2# mysql
mysql> create database test;
mysql> quit;
root@ec2# cd /usr/share/mysql/sql-bench
root@ec2# perl run-all-tests --dir='/root/'


For EBS tests I've done the following:
-created 1GB EBS volume in scalr
-attached it to the instance I was testing
-notice the device name (/dev/sdb for example)
root@ec2# apt-get install xfsprogs
root@ec2# mkfs.xfs /dev/sdb
root@ec2# mkdir /mnt/storage
root@ec2# cp -R /var/lib/mysql /mnt/storage/
root@ec2# chown mysql:mysql -R /mnt/storage/mysql

-edit /etc/mysql/my.cnf and change datadir from "/var/lib/mysql" to "/mnt/storage/mysql"
-restart mysql server and start the tests:

root@ec2# /etc/init.d/mysql restart
root@ec2# mysql
mysql> drop database test;
mysql> create database test;
mysql> quit;
root@ec2# cd /usr/share/mysql/sql-bench
root@ec2# perl run-all-tests --dir='/root/'


Instances types used and their codes:

m1.small(0.10$/hour) - Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

m1.large(0.40$/hour) - Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

c1.medium(0.20$/hour) - High-CPU Medium Instance 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of instance storage, 32-bit platform

c1.xlarge(0.80$/hour) - High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

EC2 Compute Unit (ECU) – One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.


seconds usr sys cpu tests
m1.small 1823 196.54 28.66 225.2 3425950
m1.small+ebs 1646 197.18 29.61 226.79 3425950
m1.large 1072 157.06 26.97 184.03 3425950
m1.large+ebs 1088 154.23 25.23 179.46 3425950
c1.medium 902 131.18 25.63 156.81 3425950
c1.medium+ebs 901 130.76 28.84 159.6 3425950
c1.xlarge 704 123.31 32.8 156.11 3425950
c1.xlarge+ebs 781 121.02 29.52 150.54 3425950

Bellow you can see a nice chart with how much time it took for each instance to finish the benchmark tests. Either I did something terribly wrong or EBS doesn't improve MySQL performance.

Small benchmark using ab on ec2 instances

I've performed a few small benchmarks on EC2 recently on m1.small and c1.medium using ab(Apache HTTP server benchmarking tool). The command used was:
ab -n 1000 -c 10 localhost/
n is the number of requests
c is the number of concurent requests

I've used localhost to measure the performance of the instance without taking into consideration the bandwidth.

The image used was ami-bac420d3 aka scalr app, 32 bit machine.

m1.small gave a very bad result, only 6-8 requests/second.
c1.small gave somewhat a better result, but still a long way to go... 28-30 requests/second.
On a production server, which already had traffic on it I get somewhere around 60 requests/second.

As you can see m1.small is good only for playing around with Amazon service, but not for real stuff.

I know there are a lot of things that can be done to improve performance and so on, but just wanted to show you all some results.

The Melting Pot: Introduction

Yes know what the expressions "melting pot" means. I've read on wikipedia. It usually refers to the US and their multinational and multicultural society. Whatever.

On my blog I will use this expression as a prefix for a series of posts that will follow. It will designate those articles not related to the sys admin nature of this space. Think of it as a personal mark.

Useful Firefox add-ons

This is a list of add-ons I use daily:
Firebug for Web Development
A must add-on for anyone that is doing web development.You can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page...

Nagios Checker for server monitoring
The statusbar indicator of the events from the network monitoring system Nagios. What else can be said?

GMail Notifier for email alerts
Since I use Gmail as mail solution for my day-to-day work this is a great add-on. It supports multiple accounts.

Delicious Bookmarks for keeping your bookmarks in one place
Since I use multiple desktops, laptops and other devices I hate to have separate bookmarks for each of them. delicious is a great way to keep them in one place. You can keep private bookmarks if you want.

Funny article about sysadmin

I was wandering around the Internet and I found an interesting article called "Silicon Valley Users Guide: How do I get my sysadmin to do anything?"

While it made me smile a lot I must say that most of the things pointed out there are true. This is my preferred part:

Don't question what he does all day. Systems administrators are like firemen and cops. If you don't have a couple of bored ones hanging around, you'll be sorry when there's an emergency.

Keep that in mind!

Link to the original post: How do I get my sysadmin to do anything?