Get A Sys Admin

sysadmin linux open source centos redhat rhel ubuntu apache mysql amazon ec2 s3 tutorial

Jan 30, 2010

mEgo is sky high

Yes ladies and gents, it is true! mEgo Inc, my former company where I have worked for the last 2 years as sysadmin, made the move to Amazon Elastic Compute Cloud (aka EC2) today!!!11oneone It's a moment of great joy to see this thing happening now at the beginning of 2010.

Best wishes to the mEgo team!

LE: Proof
PING mego.com (67.202.39.53) 56(84) bytes of data.
64 bytes from ec2-67-202-39-53.compute-1.amazonaws.com (67.202.39.53): icmp_seq=1 ttl=48 time=10.6 ms

Labels: , , , , , , ,

Feb 13, 2009

More Apache benchmarks using EC2 instances

Back with more benchmarks ... This time I've tested several EC2 instances, using the AMIs provided by scalr for application roles(app and app64). Site used for testing was mEgo.
c1.medium results:
Concurrency Level: 200
Time taken for tests: 311.715596 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 306810000 bytes
HTML transferred: 303610000 bytes
Requests per second: 32.08 [#/sec] (mean)
Time per request: 6234.312 [ms] (mean)
Time per request: 31.172 [ms] (mean, across all concurrent requests)
Transfer rate: 961.19 [Kbytes/sec] received

m1.large results:
Concurrency Level: 200
Time taken for tests: 420.241673 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 308313369 bytes
HTML transferred: 305097689 bytes
Requests per second: 23.80 [#/sec] (mean)
Time per request: 8404.834 [ms] (mean)
Time per request: 42.024 [ms] (mean, across all concurrent requests)
Transfer rate: 716.46 [Kbytes/sec] received

c1.xlarge results:
Concurrency Level: 200
Time taken for tests: 70.404865 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 306810000 bytes
HTML transferred: 303610000 bytes
Requests per second: 142.04 [#/sec] (mean)
Time per request: 1408.097 [ms] (mean)
Time per request: 7.040 [ms] (mean, across all concurrent requests)
Transfer rate: 4255.66 [Kbytes/sec] received

m1.xlarge results:
Concurrency Level: 200
Time taken for tests: 215.153753 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 308098602 bytes
HTML transferred: 304885162 bytes
Requests per second: 46.48 [#/sec] (mean)
Time per request: 4303.075 [ms] (mean)
Time per request: 21.515 [ms] (mean, across all concurrent requests)
Transfer rate: 1398.43 [Kbytes/sec] received


So to sum things up:
c1.medium can serve about 32 requests per second for 0.20$ per hour.
m1.large can serve less than c1.medium, around 24 requests per second for 0.40$ per hour ... not nice :(
m1.xlarge can serve 47 requests per second, but it will cost you 0.80$ per hour.
The champion is c1.xlarge can serve 142 requests per second at same price as m1.xlarge, 0.80$ pe hour.

I didn't include m1.small benchmarks, but from a previous post I can tell you it only server 6 requests per second, not a worthy instance.

My advice would be to go with c1.medium instances since they offer best price/performance ratio. But do the math for yourself!

Labels: , , , , , , , ,

Feb 11, 2009

MySQL benchmarks using Amazon EC2 instances

Here are some tests I've run on Amazon using AMIs provided by scalr for the mysql role. I've used the benchmark scripts supplied by MySQL located in /usr/share/mysql/sql-bench. I had to install a package before running the tests:
apt-get install libdbd-pg-perl

After that everything was simple:
root@ec2# mysql
mysql> create database test;
mysql> quit;
root@ec2# cd /usr/share/mysql/sql-bench
root@ec2# perl run-all-tests --dir='/root/'


For EBS tests I've done the following:
-created 1GB EBS volume in scalr
-attached it to the instance I was testing
-notice the device name (/dev/sdb for example)
root@ec2# apt-get install xfsprogs
root@ec2# mkfs.xfs /dev/sdb
root@ec2# mkdir /mnt/storage
root@ec2# cp -R /var/lib/mysql /mnt/storage/
root@ec2# chown mysql:mysql -R /mnt/storage/mysql

-edit /etc/mysql/my.cnf and change datadir from "/var/lib/mysql" to "/mnt/storage/mysql"
-restart mysql server and start the tests:

root@ec2# /etc/init.d/mysql restart
root@ec2# mysql
mysql> drop database test;
mysql> create database test;
mysql> quit;
root@ec2# cd /usr/share/mysql/sql-bench
root@ec2# perl run-all-tests --dir='/root/'


Instances types used and their codes:

m1.small(0.10$/hour) - Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

m1.large(0.40$/hour) - Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform

c1.medium(0.20$/hour) - High-CPU Medium Instance 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of instance storage, 32-bit platform

c1.xlarge(0.80$/hour) - High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

EC2 Compute Unit (ECU) – One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.


seconds usr sys cpu tests
m1.small 1823 196.54 28.66 225.2 3425950
m1.small+ebs 1646 197.18 29.61 226.79 3425950
m1.large 1072 157.06 26.97 184.03 3425950
m1.large+ebs 1088 154.23 25.23 179.46 3425950
c1.medium 902 131.18 25.63 156.81 3425950
c1.medium+ebs 901 130.76 28.84 159.6 3425950
c1.xlarge 704 123.31 32.8 156.11 3425950
c1.xlarge+ebs 781 121.02 29.52 150.54 3425950

Bellow you can see a nice chart with how much time it took for each instance to finish the benchmark tests. Either I did something terribly wrong or EBS doesn't improve MySQL performance.

Labels: , , , , , , ,

Small benchmark using ab on ec2 instances

I've performed a few small benchmarks on EC2 recently on m1.small and c1.medium using ab(Apache HTTP server benchmarking tool). The command used was:
ab -n 1000 -c 10 localhost/
n is the number of requests
c is the number of concurent requests

I've used localhost to measure the performance of the instance without taking into consideration the bandwidth.

The image used was ami-bac420d3 aka scalr app, 32 bit machine.

m1.small gave a very bad result, only 6-8 requests/second.
c1.small gave somewhat a better result, but still a long way to go... 28-30 requests/second.
On a production server, which already had traffic on it I get somewhere around 60 requests/second.

As you can see m1.small is good only for playing around with Amazon service, but not for real stuff.

I know there are a lot of things that can be done to improve performance and so on, but just wanted to show you all some results.

Labels: , , , , , , ,

Nov 4, 2008

Scalr errors after install

After installing Scalr and adding a client I've tried to add an application to test out my setup. But at the second step I've got an alert saying:
Error Type: LoadXML
Description: Incorrect XML


A quick look at the apache log revealed the problem:
File does not exist: /var/scalr/app/www/farm_amis.xml


I thought that maybe I've missed a file so I did a svn checkout of scalr repository and tried to find the specified file:
apt-get install subversion
svn checkout http://scalr.googlecode.com/svn/trunk/ scalr-read-only
find scalr-read-only -name farm_amis.xml -type f


Nothing came up. Weird. After a bit of looking around I've found a file called farm_amis_xml.php. So in fact farm_amis.xml was only a mod_rewrite directive.

Time to fix it, enable mod_rewrite and .htaccess files for apache2:

Edit /etc/apache2/sites-available/000-default and change the lines containing AllowOverride None to AllowOverride All

Go to /etc/apache2/mods-enabled and execute the following command:
ln -s ../mods-available/rewrite.load

Restart apache2 server and everything should be ok:
/etc/init.d/apache2 restart

Labels: , , , , , , , ,

Nov 3, 2008

How to install Scalr on Ubuntu 8.10 EC2 Instance

Presentation:


If Amazon EC2 doesn't ring a bell to you, chances are that you are looking at the wrong page to find solutions for your problems. EC2 stands for Elastic Compute Cloud and it's a service offered by Amazon. I will not enter in details about the advantages of using it, since this is not the scope of this post. You can read more about it here: http://aws.amazon.com/ec2/

Scalr is a fully redundant, self-curing and self-scaling hosting environment utilizing Amazon's EC2. You basically can build farms of Amazon's instances that can do load balancing using nginx, serve web pages using Apache 2, use MySQL master-slave servers or maybe you may want to define your own roles.

The beauty of this is that you don't have to monitor the health of your server infrastructure, scalr will do it for you. If a node type gets overloaded scalr will launch another instance to spread the load and the cluster will be reconfigured.

Preparing:


Generate a new key for scalr instance:
ec2-add-keypair scalr-keypair > id_rsa-scalr-keypair
Edit id_rsa-scalr-keypair so it begins with
-----BEGIN RSA PRIVATE KEY-----
and is terminated with
-----END RSA PRIVATE KEY-----
Make sure you have the correct permissions for this key:
chmod 600 id_rsa-scalr-keypair

If everything went ok you should see your new key when executing
ec2-describe-keypairs

Choosing the right AMI:


For the instance we will be using ami-7806e211 which is an AMI containing a base install of Ubuntu 8.10 Intrepid Ibex Release. More details here.

Start the instance:


ec2-run-instances -z us-east-1a -k scalr-keypair ami-7806e211
You will get some output, look for the line that begins with INSTANCE and write down somewhere the id of the instance (i-XXXXXXXX) and the address of the instance(ec2-XX-XX-XX-XX.compute-1.amazonaws.com). The status of your instance should be pending.

Check in a couple of minutes the status of your instance:
ec2-describe-instances i-XXXXXXXX
When the status is running it means that your instance is ready for work. You should have at least ssh and web ports open(22 and 80). If you are not sure execute the following commands:
ec2-authorize default -p 22
ec2-authorize default -p 80


Now connect to your instance using ssh:
ssh -i id_rsa-scalr-keypair -v ec2-XX-XX-XX-XX.compute-1.amazonaws.com
First time you connect you will be asked if
Are you sure you want to continue connecting (yes/no)?
Type yes and you should be the happy owner of a fresh Ubuntu Intrepid Ibex instance.

Update your system now:


apt-get update
apt-get upgrade


After the update is completed, logout and reboot your instance:
ec2-reboot-instances i-XXXXXXXX

Installing required software:


Reconnect to your instance and install MySQL server and php extensions:
apt-get install bind9 mysql-server mysql-client apache2 php5-cli libapache2-mod-php5 php5-mysql php5-mcrypt php5-mhash

When you install MySQL server you will be prompted to setup a password for the root account. Don't forget it, you will need it. Also you will have to restart Apache2 server after you finish installing everything, like this:
/etc/init.d/apache2 restart
You could also download their php script that checks if your system has all the prerequisites
wget http://scalr.googlecode.com/files/checkenvironment.php
mv checkenvironment.php /var/www/
chmod a+r /var/www/checkenvironment.php

Now point your browser to http://ec2-XX-XX-XX-XX.compute-1.amazonaws.com/checkenvironment.php and see if everything it's ok.

Most likely you will get only these 2 errors:
Errors:
• Cannot find SSH2 functions. Make sure that SSH2 Functions enabled.
• Cannot find SNMP functions. Make sure that SNMP Functions enabled.

Here is how to quick fix it:
Adding SSH2 support to PHP5, better known as: why don't we have php5-ssh2?
apt-get install php5-dev php-pear libssh2-1 libssh2-1-dev
Thought it will be easy? Not so quick. Try to install it with:
pecl install ssh2 "channel://pecl.php.net/ssh2-0.10"
I got an error saying:
ERROR: `make' failed
Great! Let's fix that stupid error. Edit the file /tmp/pear/download/ssh2-0.10/ssh2.c and replace the line containing:
#if LIBSSH2_APINO < 200412301450
with
#if false
Go to directory /tmp/pear/download/ssh2-0.10/ and compile the stuff manually:
make && make install
echo extension=ssh2.so >> /etc/php5/apache2/php.ini


I don't get it why they don't fix this thing. A lot of people are having this problem and are complaining!

Luckily for you and me snmp is a breeze, it is already in repositories:
apt-get install php5-snmp

Restart apache server and check now if you have all the required extensions for scalr. You should have them.

Getting the latest version of Scalr:


At the time of writing this article latest version was 1.0 RC2
Go to Scalr download page and copy the link to the latest release. Download it using wget:
wget http://scalr.googlecode.com/files/scalr-1.0RC2.tar.gz
Extract it:
tar zxvf scalr-1.0RC2.tar.gz

Create database for scalr and import the sql:
mysqladmin -p create scalr
mysql -p scalr < scalr/sql/database.sql


Put the scalr application in /var/scalr and change permissions as suggested in the spartan documentation of scalr:
mkdir /var/scalr && cp -R scalr/* /var/scalr/
chmod 777 -R /var/scalr/app/cache /var/scalr/app/cron/cron.pid /var/scalr/app/etc/.passwd


ATTENTION: I'm not planing on using this instance for anything else except scalr. Also this is a clean install so I don't have anything of interest in /var/www. Read carefully the following first line:

rm -rf /var/www/
ln -sf /var/scalr/app/www /var/
chmod a+rX -R /var/www


Edit the file /var/scalr/app/etc/config.ini and update it to your values:

[db]
driver=mysql ;Actually mysql is the only option here - mysqli doesn't support nconnect(), which is essential for PCNTL (which is essential for crobjobs)
host = "localhost"
name = "scalr"
user = "root"
pass = "*YOUR PASS HERE*"

TO DO: make a mysql user for scalr.

Put your EC2 access certificate into /var/scalr/app/etc/cert-XXXXXXXXXXXX.pem
Put your EC2 private key into /var/scalr/app/etc/pk-XXXXXXXXXXXX.pem
ATTENTION: This part is a bit tricky. If you don't put the right settings you will not be able to start instances. I warned you!
Login to Amazon AWS and go to Home->Your Account->Access Identifiers
Go to http://ec2-XX-XX-XX-XX.compute-1.amazonaws.com/ and login with admin/admin
Go to Settings->Core Settings. Modify the following fields:
Password: duh!!! change it!
Email: your email address here
Account ID: AWS Account Number, it's called Account Number in AWS and it's top right. Remove the '-' from the number
Key Name:Scroll down in AWS untill you see Your X.509 Certificate: Copy everything between 'cert-' and '.pem'. Also XXXXXXXXXXXXXX is the string in the filenames of cert-XXXXXXXXXXXXX.pem and pk-XXXXXXXXXXX.pem. If they don't match you will have problems.
Access Key:Look for Your Secret Access Key: in AWS and click on Show. Copy the string.
Access key ID: is Your Access Key ID: in AWS.

Hit save.

setting crontab:


Type crontab -e and add the following lines to cron:
* * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --Poller
1 1 * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --RotateLogs
*/15 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --MySQLMaintenance
*/6 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --DNSMaintenance
*/3 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --DNSZoneListUpdate
*/2 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --DBQueueEvent
*/11 * * * * /usr/bin/php -q /var/scalr/app/cron/cron.php --Cleaner

You are done. I hope.

How to terminate the instance:


I thought to write down instructions on how to terminate an instance. You should know how, but just in case, here it is how to stop the instance forever and not pay for it anymore. ATTENTION: terminate will really delete the instance so there is no way you can reconnect to it or recover it. Double check what instance you terminate!
ec2-terminate-instances i-XXXXXXXX

Labels: , , , , , , ,