I’d initially assumed that when a Rails application (lets say Mongrel) was sending a file with the ’send_file’ method then it couldn’t handle other requests as they came in.  Seeing as that was an assumption rather than fact I setup a download action on my Ubuntu dev server to show this happening.  Basically a website supported by a single Mongrel – if it’s busy sending a large file additional requests will get queued up eventually giving 503 errors I figured as they timed out.

I set the action going to download a 40mb test file on the server, opened up another tab and loaded another test page and it appear straight away.  So my assumption was wrong, but not completely wrong it turns out.  Mongrel is still sending the file though as if I kill it the download terminates.  This is because Mongrel has it’s own internal request queue and only sends one request at a time to your Rails code. Hence the ability to handle additional requests while someone is downloading a file and websites running clusters of Mongrels.  Getting Mongrel to send the file isn’t the best use of resources though.

For Nginx to send files on Mongrel’s behalf two changes are needed.  Firstly you need to tell Nginx that it should be doing this and from where.  The Nginx sendfile page is quite helpful in this regard, you’ll end up having something like this:

location /files/ {
  internal;
  root
/;  # note the trailing slash
}

Note, sendfile is enabled by default in all nginx.conf files I’ve seen.

Secondly, in your Rails download action do something like this:

if
  head(:x_accel_redirect => "/files/#{filename}",  :content_type => File.mime_type?(file), :content_disposition => "attachment; filename=#{filename}")
else
  send_file("#{RAILS_ROOT}/files/#{filename}", :type => File.mime_type?(file))
end

It is important that you specify the correct mime_type to stop the receiving browser guessing and potentially changing the file extension.  If one of the standard rails attachment gems is being used, then you’ll likely have that information already.  But if you don’t, like in the example above, then mimetype_fu is a very handy plugin as it extends the File class by adding the mime_type? method.

If you store files with guid like names, then the file name received by the user can be controlled by changing the :content_disposition value.

Finally, the reason I’ve specified the source tree root is so that files from multiple top level folders can be accessed from the one location entry in the Nginx config.

There is a plugin somewhere that does the head changes for you, but it’s been abandoned and even has a message suggesting that it shouldn’t be used for that very reason.  I’ve not used it for that reason and because it assumes all file downloads are to be handled by Nginx.  That’s a nasty assumption that would bite someone one day, so not very fun.  So I’m in the process of rolling my own.

Apache can do the same with with mod_xsendfile.

No related posts.

The page I’ve linked to above is worth reading as it covers performance and durability quite succinctly.  It’s pretty awesome to learn that you can just treat each block (EBS) as you would traditional physical storage, striping multiple mounts to achieve greater performance.  Hurrah!  Given the only different in cost of having three 30gb blocks striped compared to a single 30gb block would be three times the access costs I wonder if this would be the more common style of configuration?  I also wonder how backups / snapshots work in this situation as synchronization would be quite important one would think.

I find sometimes it’s the case that numbers quoted by a merchant aren’t often attained in the real world as they only reflect usage under ideal conditions.  Internet speeds in New Zealand, pictures of fast food and cosmetic benefits being good examples.  So I wonder if the same will apply to this new service?  Time will tell and I’m sure it’ll be big news if Amazon’s numbers are considerably off the mark.

Come to think of it I don’t know how Amazon’s existing services stack up against what is advertised.  I’m gonna find out though as I intend to see how much I can squeeze out of their service too.  Apart from the geek factor, I’m doing this as clients are increasingly asking about Amazon EC2 after I normally recommend either Slicehost or a dedicated box somewhere.  Basically at the moment I don’t know, when I need to know.

So I have four big questions around using Amazon EC2.  Firstly a big cause of concern for me is their service having already experienced a number of notable down times including what appears to be reported data loss.  With their latest significant event Amazon has been very open about the problem and what they are doing as a result – nice and positive.  Although, as nice as that is, they should really stop having them.

Secondly is performance, obviously.  I’m gonna put everything that can’t be replicated on EBS, including all logs that interest me.  Of note, I’ll have a webserver, a load balancer and many many application instance continuously logging away from multiple servers and having a good time about it.  Oh and also the database.  Performance better not blink.  Previously a big turn off for me has been the situation of ‘what logs?’ -  if an EC2 server suddenly stops, you have no way of knowing what just happened.  Yeah I know about services like RightScale that attempt to minimize this, but its not good enough by my standards.  Plus their sign-up fee is a big turn off to me, presumably they are trying to protect their IP from free access.  Anyway, who does big sign-up fees anymore???  It’s so 2005.

Thirdly is unexpected restarts.  What event based actions can I automate?  I don’t know and this is possibly just because I’ve not dug into it deep enough yet.  When they restart, EC2 servers are blank again.  Can I set them to be automatically loaded with a script which reverts their state back to what it was pre-reboot?  I doubt this would be as quick as I’d want so I could just symlink most of the OS to EBS – depending on it’s performance.  It implies that either Amazon provides a harness that sits around all your servers for kicking off the scripts as needed, or you use a third party or you roll your own.  I’ve been looking out for an excuse to setup a big kickass high availibilty setup, then it wouldn’t matter!  Hmmmm.

UPDATE: You can mount EBS as the file system on EC2 as noted here.  That basically implies it’s up there in the performance stakes and obviously no need to reinstall everything after a reboot.  You can also create a new EBS from a snapshot which is extremely handy.

Forthly, permanent ip addresses – I know they have them but again that’s about the limit of my knowledge.  The lack of an EBS like service has stopped me seriously investigating AWS until now.  Can I have more than one ip address per server?  Can I have a floating one if I want to roll my own my high availability configuration?

Backing up user content and other important data is a really big deal for a website, especially as it starts accumulating and accumulating.  I have found Amazon S3 is my best mate for this – especially for doing regular rsync like backups.  If a website is entirely hosted with EC2 and EBS this becomes a whole lot easier to the point of being stupidly easy.  That unto itself is a really big deal.

At the moment all server oriented virtualized services that I can recall using and reading about are essentially just replications of physical devices – normally with marginally lower performance characteristics while being more reliable due to their redundant nature.  I do wonder though when someone will come up with something new that’s not available in the physical world and what that will be.

Aside from the concerns I’ve expressed above, I’ll just add that what an awesome learning tool AWS and like services are becoming.  Who cares if you screw up – learn, wipe and start again.

I’ve got even more geeking out ahead of me now, sweet.

No related posts.

git pull
mongrel_rails cluster::restart

All my server is showing at the moment is this.  It’s amazing I know and people will talk about it for years to come.  Probably get a couple of book deals out of it too I expect.

Basically, I don’t have anything else to put there at present. I have two website ideas under consideration though, and a third thought of moving this blog to my own server as it’s not particularly speedy here at Dreamhost.  You get what you pay for though…and I’m not paying much.  Dreamhost are good, don’t get me wrong on that.

Ideally I’d want to use some Ruby on Rails blogging software but a quick google doesn’t turn anything awesome up.  I’d probably have to reproduce all the extra plugins I’m using etc, so I likely be sticking with WordPress for the time being.

Anyway, I’ve got Apache installed and configured with a really bare bones script:

NameVirtualHost *:80
 
ServerName <your_domain_name>
 
<VirtualHost *:80>
  ServerName <your_domain_name>
 
  DocumentRoot <your_code_path>
 
  <Directory "<your_code_path>">
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
  </Directory> 
 
  Alias /images <your_code_path>/public/images
  Alias /stylesheets <your_code_path>/stylesheets 
  Alias /javascripts <your_code_path>/javascripts
 
  ProxyPass /images ! 
  ProxyPass /stylesheets !  
  ProxyPass /javascripts !  
 
  # Check for maintenance file and redirect all requests
  #  ( this is for use with Capistrano's disable_web task )
  RewriteCond %{REQUEST_URI} !\.(css|jpg|png|htc|gif)$
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteCond %{SCRIPT_FILENAME} !maintenance.html
  RewriteRule ^.*$ /system/maintenance.html [L]
 
  # re-write so that all non-static requests go to the cluster
  RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
  RewriteRule ^/(.*)$ balancer://mongrel_clusters%{REQUEST_URI} [P,QSA,L]
 
</VirtualHost>
 
<Proxy balancer://mongrel_clusters>
  BalancerMember http://localhost:1234
  BalancerMember http://localhost:1235
</Proxy>

It’s fairly basic, if the requested file isn’t found on disk pass it off to the mongrel cluster, oh and don’t ever ask mongrel for a javascript, stylesheet or image file.

Obviously this requires that I have mongrel installed:

gem install mongrel_cluster

As indicated above I’ve already gotten the source code where I want it on the server, I’ve stubbed out a bit of the Capistrano style directory structure.  So now I can just run my cluster of two mongrels and all just works fine and dandy.

Next up I’ve got to create a good page for benchmarking against, and then do just that before I start upgrading the configuration and code to make it super fast.

No related posts.

By using threads to serve requests, it is able to serve a large number of requests with fewer system resources than a process-based server.

Sounded good to me when I first read about it.  My experience with both worker and pre-fork has shown this to be very true, in particular the improved resource usage – hence my choice for with my 256mb slice.

I’ve not used the newer, and shouldn’t be used for production, Apache MPM event.  I do plan to read up more about it and then give it a whirl at some point though.

I had been using the following command to start and stop Apache, but found it to be woefully limiting:

/etc/init.d/apache2
* Usage: /etc/init.d/apache2 {start|stop|restart|reload|force-reload|start-htcacheclean|stop-htcacheclean}

Instead I’m now using the alternative that I went looking for:

/usr/sbin/apache2ctl
* Usage: /usr/sbin/apache2ctl {start|stop|restart|graceful|graceful-stop|configtest|status|fullstatus}

Which as you can see actually has the status command!  Why this isn’t in the /etc/init.d based script I don’t know.  Maybe I’m shouldn’t be controlling Apache through init.d, but lots of others seem to be…  Anyway as well as the status command I was lacking the graceful command which I find invaluable.

Both the /etc/init.d/apache2 and /usr/sbin/apache2ctl scripts are compatible with each other as the former just calls the latter.

Mod_status is required for the status command to work, which is fine until I install my rails app as it will respond to /server-status instead.  As a result I’ve reconfigured mod_status to run on a different port and while there I setup mod_info too.  I obviously had to punch a new hole through the firewall so I could access this off-server.

Finally if you get the following error when starting and stopping your Apache server:

Could not determine the server's fully qualified domain name, using 127.0.0.1 for ServerName

You need to define ServerName somewhere in addition to within a virtualhost area.  This is dicussed here.

No related posts.

The one thing to note here is that by running the automated self update, RubyGems effectively downloads the source to the latest version and compiles and distributes it onto the server.  This breaks my previously mentioned rule of don’t compile source, but in this case I have no choice as this is how RubyGems works.

Since this leaves me with two seperate installations, the apt-get package and the compiled version I then uninstalled the package.

apt-get remove rubygems libgems-ruby1.8

Before I downgraded back to a 256mb server, ie while I still had twice the grunt, I installed Ruby on Rails.

gem install rails

Noting I’m still logged in as root at this stage so I don’t need to pre-fix the above command with sudo.

Next up, MySQL.

apt-get install mysql-server mysql-client libmysqlclient15-dev

You’ll get asked for root’s password as part of the package’s installation process.  Do use one and don’t loose it.  I use passphrases rather than passwords as they’re way more secure and I can easily memorise them.

Following this I then logged into MySQL and created the main database, and while I was there I setup the rails user account.  As you can see I’m being pretty liberal with the access rights at this stage.  That’ll change though.

CREATE DATABASE performance_prod;
CREATE USER 'rails'@'localhost' IDENTIFIED BY 'some_passwphrase';
GRANT ALL PRIVILEGES ON performance_prod.* TO 'rails'@'localhost';

At this stage I’m using the vanilla MySQL config meaning the database will only respond to requests from localhost.  I’ve infact not looked at the default MySQL config for Ubuntu yet, but if it’s anything like the default one on CentOS it’ll be getting some love real quick.  That one assumes you only have something like 16mb of ram for the whole system!  Changing it makes a big difference.

As I’ll be wanting my rails app to talk to MySQL I’ll be needing the API module that allows this.  Under CentOS and OSX this is a gem that can be particullarly annoying to install, especially under OSX.  Thankfully someone has kindly packaged it up for us.

apt-get install libmysql-ruby

As an aside, I bumped into this article listing ten ways to make linux boot faster and pretty much ignored them all except for the final point.  Tried it and it appears to work a treat.  Edit /etc/init.d/rc and change the following setting to:

CONCURRENCY=shell

No related posts.

A second problem with what I installed was that the command ‘ruby -v’ wouldn’t work but ‘ruby1.8 -v’ would give the desired result.  Ruby had been installed everywhere prefixed with ‘1.8′.  Makes sense as it allows you to have multiple versions of ruby installed at once, but things like mongrel won’t work as they are expecting to find ‘ruby’ not ‘ruby1.8′.  Meaning I’d have to create a bunch of symbolic links like below to solve the problem.

ln -snf /usr/bin/ruby1.8 /usr/bin/ruby

I don’t like that as it’s fails the don’t compile source rule as the moment I upgrade to ruby 1.9 I’ve got to remember to manually change all those symbolic links.  A bit more digging and I discovered that there are packages that do this for me, so in the end I’ve run the following to install ruby on Ubuntu:

apt-get install ruby libruby irb ruby1.8-dev libopenssl-ruby libreadline-ruby ri rdoc

Package dependencies mean that all the previous packages I was trying to install are also installed.  Nice.

Next up RubyGems.  RubyGems uses the yaml and zlib libraries, as noted here,  but it’s package unfortunately hasn’t had this included in it’s setup, so the following is required for an install:

apt-get install rubygems libyaml-ruby libzlib-ruby

This only installs version 0.9.4-4 though so you have to run the self update task:

gem update --system

Should be pretty simple, but thirty minutes after running this it’s still only telling me the following:

Updating RubyGems...
Bulk updating Gem source index for: http://gems.rubyforge.org

A quick google revels that this can happen in virtualised systems (ie zen which slicehost use) when you don’t give them enough system resource.  My server only has 256mb of ram and the gem process quickly ate 85% of it then stopped doing anything.

I might leave it running while I go watch the rugby.  If it’s still essentially hung after the match I’ll probably upgrade the server to a 512 instance and try again.  Then upon success, hopefully, I’ll downgrade back to 256.  Obviously in a real life produciton environment this wouldn’t be an acceptable practice, but then i wouldn’t be using a server with 256mb of ram either.

No related posts.

Can you give me a quick overview on what the trouble was running passeneger/enterprise ruby over multiple servers?  I’m half way through setting up our infrastructure with 1 load balancer passing on to 2 web servers…  running passenger/enterprise ruby.

The short answer is that there isn’t any problem – if you’re running Apache on each of your application servers.  I just don’t.

After an exchange of emails I learned that they are using the infrastructure provided for the contract being worked on, which isn’t necessarily their preferred choice.

The reason I haven’t followed the above paradigm is that Apache (or any hardened webserver) can handle several orders of magnitude more requests a second than what your average Ruby on Rails application can on the same hardware.  Say one Apache to one hundred mongrels, without being very scientific about it.  So why hamstring Apache to the lowest common dominator?  Given each instance of Apache has a minimum footprint, it’s a waste of resource.  Yes Apache also handles static files from disk that your app will never see, that still doesn’t change the equation much though.

Think of it this way, database servers get their own box early on, same concept.  Separate out the slowest beast into it’s own replicable box and just scale out from there as needed.  App servers should just have your application on them.  If you have more traffic than one Apache can handle, then you’ll obviously want to load balance to multiple web servers.

I’ve been stuck with a similar infrastructure hurdle before, in my case it was a hardware load balancer which the provider had setup stupidly.  It took them 4 days and two reboots to get it as I wanted it.  Extremely frustrating, especially as the reboots caused site downtime which I like to keep to a minimum.

For this project my intention is to performance test the updated configuration after each change and roll back when there isn’t a benefit.  I’m not going to have a hardware load balancer so I can’t really replicate the above, but I could pretend with three servers…

Anyway, I’m back online from relaxing over the weekend and sorting out what version and edition of ruby I’ll be using for my starting point.  After that I need to finish setting up the server for delivering a Ruby on Rails application and then actually create an application for deployment.  I’m open to suggestions.

No related posts.

So, anyway.  I’ve got a server up and going doing absolutely nothing, and I’m assuming you do too at this point.  It’s just waiting for me to begin steaming on ahead with the initial production stack I’m planning on using first.

Lets get Apache on there for some instant gratification:

sudo apt-get install apache2 apache2-doc apache2-mpm-worker apache2-utils apache2.2-common libapr1 libaprutil1 libpq5 libexpat1 ssl-cert

That’s all you’ll need, the ssl-cert package is there so that people can do things over https:// with your website like logging in, buying things etc….  Provided you’ve created a port 80 hole in your firewall you should be able to go have a look at the default page once you’ve made sure apache is up and running.

sudo /etc/init.d/apache2 force-reload

Next I tried getting ruby on there but discovered that it already was, but not really.  It looked like most of it’s source was, but none of the binaries were, fully weird.  So I uninstalled the base ruby packages:

sudo apt-get remove ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8

Then reinstalled everything I thought I’d need:

apt-get install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8 libreadline-ruby1.8 libruby1.8 libopenssl-ruby libopenssl-ruby1.8

Unfortunately this is Ruby 1.8.6 at patch level 111 (2007-09-24) which has some significant security issues associated with it covered last month on the rails weblog.  I was surprised to find that the ubuntu packages havn’t been updated and that a quick google didn’t turn up an immediate solution for me apart from Ruby Enterprise Edition.  I don’t really want to manually compile it from source and/or apply some back ported patch.

I’d read that you only gain real advantage from using this fork of ruby when it’s used in conjunction with mod_rails so I wasn’t going to bother with it.  I might now though, especially as they appear to have had a quick turn around on this issue.  Additionally I want to quickly move to using Rails 2.1 with Ruby 1.8.7, and again, the ubuntu package was an old build of 1.8.6.

No related posts.

The Centos image at slicehost is bare basic and only comes with openssh, where as the Ubuntu image has a bunch of things out of the box which is nice.  The down side of this being that I’ve got to evaluate and remove the things I don’t want.  I know what I’m doing with iptables, so I don’t want FireHol for example.

Seeing as my blogging on this project isn’t targeted at how to setup a server from scratch I won’t bore you with the details of why I use vim and zsh instead of something else and bash etc..  Although I will though throw in a couple of links I found helpful in my transition between distributions which answer my ‘err, how do you achieve this in Ubuntu’ and ‘where the feck do they put that’ type questions.

As previously mentioned I like to hit up HowToForge as it’s a fantastic resource and has served me well for a couple of years now.  On there people like to post instructions on setting up their idea of a ‘Perfect Server’, I’ve pulled from in a couple of places out of this one for instance.  For example, I didn’t know about FireHOL until reading this.

Additionally the community help/docs portion of the Ubuntu site was quite helpful, in particular the iptables how-to solved my problem of persistence through restarts.  I have just used my base configuration for this and will expand it out as needed.

Next up is actually installing and configuring the main stack.  I’ll go and find myself a good code formatting wordpress plug first before blogging about that.

On a side note, with my first post on this topic three days ago I mentioned Ponoko.  The post was published less then two hours before Google’s daily alert digest to me covering the key word of ‘Ponoko’.  My post was in that list, and the same happened for my next post.  I was quite impressed as to how quickly that got picked up considering how inactive my blog has been up until recently.  So I’ve mentioned Ponoko again. primarily just for the heck of it, and to get the traffic. :-)

No related posts.

Ubuntu, because basically the wealth of resources available online for this distribution is massive compared to the likes of CentOS/RedHat.  Well, the places where I look anyways.  I use SliceHost for hosting Ponoko and couldn’t praise them more.  For US$20 a month I get a whole server to myself, persistent storage and fixed ip address – I can do with it as I wish.  If Amazon AWS persistent storage was publicly available and they didn’t have spam problems I’d be considering them.

So the plan is to setup a stack for serving a Ruby on Rails application on a single server, and then to see how much I can squeeze out of it before bring in more servers/slices.  The advantage of doing this as a personal project is that I don’t have to be massively pragmatic about it – more of my decisions can be geeky based rather than asking myself ‘will this make money’?

On with the plan.  This is going to be done in stages, the first is just taking what I’ve previously implemented and moving that to Ubuntu.  Essentially a very traditional stack as follows:

Apache 2.2
Mod_Proxy
MongrelCluster
Ruby 1.8.7
MySQL 5.x
MemCached

Initially going with the above familiar stack allows me to focus my learing on the Ubuntu way of things without the distractions of setting up a new stack.  Additionally I’ll need to come up with a small set of performance tests to track my progress to prove my assumptions/choices.

Once that’s done, the intention is to upgrade the stack to the following:

Nginx
HAProxy
Thin
Ruby Enterprise (or something else maybe)
MySQL 5.x
MemCached

Astute readers will note that I’ve not included mod_rails in there.  My intention is for this configuration to become a multi-server setup supporting a fictous site experiencing heaps of traffic.  Mod_rails isn’t going to help in that situation as it’s limitied to the server that Apache is on only.  So no point learning about it.

I’ll be throwing HAProxy in there as it’s screams while the ‘fair’ load balancer for nginx apparently still has subtle problems that would be of concern to a busy live site.  I want a site that can be updated without being taken down and HAProxy will allow me to do this.

The overall goal is a high performant ruby on rails website that can scale as needed while experiencing a minimum of downtime.

No related posts.