Last year I started a personal project to up-skill in some aspects of web development I was lacking in, like Javascript.  I had an idea for a game I’d been bouncing around for several years so I created that.  One thing led to another and now it’s become quite a serious project.

The main page for ESDAO gives an overview of what it is about, you can use this invite link to take the game for a spin.

There is a small base of committed testers enjoying the game – but it’s not really growing much and I’m kinda stumped about the best way to fix that.  I have some ideas but I have little experience in this type of thing and would love to get advice from someone with a successful background doing just that.

The game itself is close to completion and my next major tasks are getting more people playing it and sorting out the look and feel.  I have a designer lined up, but have yet to commission a site design while I work out what to do next.

So, I’m after an advisor who I can meet on a regular basis.  Someone who I can pay for good objective advice.  I’m based in Wellington and would obviously prefer someone local.

If you’d like to know more about me I have a page for that.

I’m best contacted via email – nahum.wild@gmail.com

No related posts.

I was already quite busy when everything stepped up a couple of gears around February due to going into business with  ProjectX and thus my friend John Clegg.  I’m slowly taking over the back end of the business while he focuses on the front end.  Essentially I get more toys and larger projects to play with :-)

To be realistic I reckon I’ll only get out a blog entry once every couple of weeks at this stage – which will be hard as there’s an increasing amount to blog about.  Top of my list is Nginx with the recently updated Phusion Passenger and Ruby Enterprise Edition which is powering our new Redmine server.

Stay tuned.

No related posts.

So with that in mind I’ve setup a new google group, WellOptimised.  If there is enough interest I’ll organise meetings for those here in Wellington.

If you are interested pop along via the above link and introduce yourself.

No related posts.

Since I occasionally have quite a bit to say, like in my previous entry or this one about where business logic should reside, I’m now only showing excerpts on the index page rather than whole entries.  I’ve also gone though and added in a couple of helpful plugins such as auto suggesting related posts and easy social bookmarking.  My next step is to change the source code syntax highlighting css – I’ll probably just use the Textmate bundle in Textmate.

Currently the homepage ySlow score is 79, I can get this to 85 if I concatenate and minify the css as doing this reduces size and number of requests for empty caches.  This hasn’t happened yet as it’ll require a bit of hand tweaking due to not working out of the box.

Overall I’ve gone for a more clean, simple and wider look by using Acamas which is based on the opensource Thematic framework.  It looks great on a Mac, pretty good under Windows and terrible in IE6.  If you experience any problems, in particullar with the rss feed (now thru feedburner.com) please drop me a line.

Update: I’ve gotten the ySlow score up to 90 now as the css is contenanted and minified plus the sidebar images are now being hosted locally allowing proper expires headers.  With the css files I’d simply been adding them together in the wrong order.  There is still a problem with css expressions though, but this will go away when I drop the syntax highlighting plugin – further boosting my yScore.

No related posts.

The problem was this: somewhere along the line attachment_fu was updated to execute the following line of code on classes using the has_attachment directive:

attr_accessible :uploaded_data

This is taking advantage of a security feature in Rails, it instructs the model to only allow :uploaded_data to be set through update_attributes and any like methods such as new and create.  So whitelisting bulk settable attributes.  It doesn’t raise any errors, just warnings in your log like this:

WARNING: Can't mass-assign these protected attributes: <attribute_name>

Due to no errors being raised the problem was presenting itself in other parts of the application as expected data was not being found.  Fortunately I was tailing my dev log and noticed the above warning message.

The problem in this situation was the historic code creating instances of the model by passing a hash through to SomeModel.create.  As this area of code was not utilising data posted to an action it was not a security risk.  Very fustrating, especially as this was common practice throughout the codebase.

As I didn’t want to do a huge bunch of re-keying my initial solution was to comment out the offending line in attachment_fu.  Yes this is a big nasty fix but there was time pressure, other problems to solve and I didn’t want to be held up having caused an unnecessary amount of manual testing.  In case it was forgotten about  and overriden with a future plugin update, a quick test with big comment was added.

This project hadn’t updated it’s gems and plugins for a while, meaning it was a big shock to the system when they where.  This was the biggest headache, all other problems were mainly just gem interfaces changing, so a easy search and replace matter.

I like passing a hash to mass-assign data in Rails as well as the obvious coding benefits it’s visually more readable too, but this is out weighed by my great dislike of having my sites hacked into.  So it’s a shame things have had to go this way.

No related posts.

I’ve done this mainly to stop paying for hosting twice, but also because if I’m going to be talking about performance I can’t possibly have a slow blog and my options were limited with the Dreamhost account.

My immediate problem being that I wanted to move it straight to being served by Ngnix rather than Apache – not something that Wordpress is setup for.  After a bit of googling I found that someone had solved all the problems I was going to have for me!  This post has blow by blow instructions for setting up a wordpress blog on Slicehost and Ubuntu, I couldn’t have found anything better.

It worked exactly as advertised, but me being me I could help tweaking and updated the nginx site config to include expire header for static assets, like so:

server
  {
 
  listen   80;
  server_name www.motionstandingstill.com;
 
  root   <path_to_site>;
 
  access_log <custom_log_path>/access.log;
  error_log <custom_log_path>/error.log;
 
  location ~* i.+\.(css|js|jpg|jpeg|gif|png)$
    {
    expires      7d;
    }
 
  location /
    {
    index  index.php index.html;
 
    # Basic version of Wordpress parameters, supporting nice permalinks.
    # include /etc/nginx/conf/wordpress_params.regular;
    # Advanced version of Wordpress parameters supporting nice permalinks and WP Super Cache plugin
    include /etc/nginx/conf/wordpress_params.super_cache;
    }
 
  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  location ~ \.php$
    {
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_index index.php;
    include /etc/nginx/conf/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME <site_path>/public/$fastcgi_script_name;
    }
  } # server

Later I found someone had posted a different conf with easy support for multiple blogs, but I’ve not moved to using it as what I’ve got works.  What it also does is add the expire clause support like I have done above.

When moving this blog to Slicehost I initially set it up on a temp subdomain to make sure it was working sweet first and because I wanted to try out some different themes and play with a couple of other things.

I quickly discovered that as it was running on a different doman than it was configured for, Wordpress would redirect my browser to the correct domain meaning I couldn’t get into the new site.  Ugh.  So I had to whip out my sql skills and manually update the appropriate settings.

UPDATE wp_options SET option_value = 'http://temp.motionstandingstill.com' WHERE option_value = 'http://www.motionstandingstill.com';

I actually used a different domain name and explicitly updated two rows rather than the generic ‘where’ clause I have, but you get the idea.  This allowed me in which is what I wanted.

Then I took the opportunity to tidy up what plugins I’m using as I’d gotten a bit ‘plugin install happy’ when first setting the blog up.  As well as getting rid of unwanted plugins I installed two new plugins I’d recently found, the first a redirection plugin with regex support and the second a security checking plugin.

Once I’d got everything how I wanted it, just sticking with the same theme (the misty tree photo is mine), I grabbed a final db backup from Dreamhost and restored it on Slicehost then changed my DNS A record.

All in all, much easier than I thought it would going to be thanks to some helpful results from google. :-)

Update: Slicehost has been aquired by Rackspace.

No related posts.

For me the reasoning is distilled down to my programming style.

Pragmatism

When it comes down to it, what vital benefit does it give you over :dependent => :destroy or model validation?  Will this give you increased quality faster, quicker and cheaper?

Premature Optmisation

Basically don’t waste time optimising something that isn’t slow or is hardly used.  Rails is so slow that some fractional speedup on the DB isn’t really going to be noticeable by the end user.  Instead examine your site with NewRelic or FiveRuns and fix your bottleneck code – way more bang for your buck.

Data is Valuable

So don’t go around deleting it!  If a user wants to delete some data you should want to know why.  Looking at the data can aid in determining why without having to bug the user about it.  Yes there are occasions when this does need to happen, but they should be the exception to the rule and thus explicitly coded for.

Just mark a record as deleted and filter it out automatically, that’s easy enough to do with rails – way easier than any other language I’ve used before.  Better yet, search github and use someone else’s wheel.

Convention Over Configuration

Basically put, you do things the way that everyone else has done it.  Walk the well walked path so to speak’ and you won’t experience any bizzaro problems that someone else hasn’t already solved before you.  As you’ve created your app in a predictable way, someone else can easily come in and contribute to your project, or visa-versa and it makes the whole Rails community more accessible to you.

So in following the conventions and as you are not encouraged to use foreign key constraints, Rails isn’t expecting data to be deleted out from underneath it.

Don’t Repeat Yourself && Model View Controller

DRY is another of the core philosophies behind Rails – partials are a real good example of this especially when you start to think of them as essentially methods.  On top of the obvious ’speed to deliver’ benefits this concept provides, it also causes you to distill your code down to being more readable and accessible.  This ties in nicely with MVC principles as you then tend to group like logic together which is what MVC is all about.

Foreign key constraints are business logic as they describe and implement a behavior – “when you delete this record, also delete these other records over here”.  So why would you go and put some ‘important to know about’ business logic in an isolated and often hard to reach place?

Documentation

The one great thing I learned from Steve McConnell’s Code Complete was how to write comments, they should describe the intention of the code, rather then repeat the code.  This nicely ties in with the DRY principle.

I’m also a great believer in self documenting code, Rails and Ruby help heaps with this, and generally write comments to help simplify complex code.  Instead of my code being littered with redundant comments cluttering everything up, it just contains important ones as they are more likely to be read and updated when the code is next worked on.

If you use foreign key constraints in your DB you’ll likely end up documenting them (ie repeating them) somewhere more accessible, in the models being an obvious place.  That wouldn’t be because the business logic lives there would it?.  Have a think about that.

The Rails Way

Looking at the Convention over configuration, Don’t Repeat Yourself and Model View Controller concepts above – as much as Rails is a framework it’s also about a way of doing things.

Over the years I’ve seen a notable number of people, new to rails and these concepts, trying to make rails work their way – often with less than desirable results.  If you embrace Rails and it’s ‘way’ then it works beautifully and is a pure joy to use, rather than a frustrating experience.

For me

Coding is an art form and as such a creative process.  If something doesn’t look, or even feel right then it’s wrong.  In the end foreign key constraints on a web app just rub me up the wrong way.

No related posts.

Problems arose once I rebooted it without the monitor plugged in as it defaulted to 800×600 without any obvious way of controlling this. Things got real annoying though when I discovered that a number of dialogs are longer than 600 pixels and I couldn’t see half the windows. That’s a real basic thing which shouldn’t be wrong.

So I tried installing the VNC Server itself and running that with options specifying my desired resolution

sudo apt-get install vnc4server
vncserver :1 -geometry 1024x768 -depth 16

only to be presented with X11 rather than the Gnome Desktop Manager when I connected. Of note is that when you first run vncserver it’ll ask you for a password and setup a ~/.vnc folder to store that and an xstartup script.

After a bit of a google I found a helpful post showing how to configure and launch vncserver through xinetd as well as powering up GDM. Follow it exactly as it works a treat. It’s part of a thread titled HOWTO: Set up VNC Server with resumable sessions.

Logging in though reveled a new problem – the Gnome Theme Setting manager was constantly crashing and causing the current theme to continually be applied then dropped. I was given this error “there was an error starting the GNOME settings daemon“.

Turns out this is a recent problem with the 4.1.1 version of vnc4server and the fix has yet to be rolled into a package for Ubuntu. I found custom deb files with the latest version of vnc. Francesco Santini has helpfully copied across the compiled binaries from the latest vncserver 4.1.2 Fedora 9 packages, which have the fix, into has own i386 deb files for people to download and install – no help for my 64bit install of Ubuntu though.

I actually tried looking for the vnc source code, thinking I could just compile that directly, but that route started proving harder than I thought it should. So, I’ve ended following Francesco’s lead and copying out the binaries from the Fedora VNC x86_64 packages manually like this:

wget ftp://rpmfind.net/linux/fedora/updates/9/x86_64.newkey/vnc-server-4.1.2-31.fc9.x86_64.rpm
wget ftp://rpmfind.net/linux/fedora/updates/9/x86_64.newkey/vnc-libs-4.1.2-31.fc9.x86_64.rpm
 
rpm2cpio vnc-server-4.1.2-31.fc9.x86_64.rpm | cpio -dimv
rpm2cpio vnc-libs-4.1.2-31.fc9.x86_64.rpm | cpio -dimv

And then copy them over top of my existing vnc binaries like so:

sudo cp usr/bin/* /usr/bin
sudo cp usr/lib64/xorg/modules/extensions/libvnc.so /usr/lib64/xorg/modules/extensions
sudo cp usr/lib64/librfb.so.* /usr/lib64

If you don’t have ‘rpm2cpio’ you’ll have to install rpm itself, a bit unnecessary and self defeating sounding I know:

sudo apt-get install

I found the latest vnc-server and the vnc-libs Fedora 9 x86_64 packages on rpmfind.net.

After doing all that I seemed to have made the situation worse and was getting a connection denied error from my viewer. A but more hunting and the problem turned out to be that the Fedora binaries, Xvnc in particullar, were expecting the libcrypto package and for one of it’s binaries to be differently named. I figure that’s due to them being compiled for Fedora. The following sorted that problem:

sudo apt-get install libcrypto++7 libcrypto++-utils
sudo ln -snf /usr/lib/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.7

From memory I don’t think vncserver had this problem, just Xvnc, making the issue a bit harder to diagnose as running vncserver manually allowed me to connect, but not with the GDM login.

Having done all that I can now login to GDM and don’t get the crazy theme errors anymore, plus it’s noticeably faster. Something I discovered is that Chicken of the VNC doesn’t disconnect when you close the viewer window – and thus you can’t reconnect until you have quit the application. A bit suck, but oh well.

During my travels I found that I could manually do what xinetd was doing with this command:

sudo vncserver :5 -geometry 1200x750 passwordFile=~/.vnc/passwd -query localhost -fp /usr/share/fonts/X11/misc

Additionally I discovered the ‘alien’ package which converts rpm files to deb, which might ultmately be an easier way to install the latest version of vncserver from Fedora while maintaining a single package system.

No related posts.

Put basically if you call count on a relationship it will always make a db call to see how many there are even if the relationship has been populated.  If you use size or any? they will too, unless the relationship is populated, in which case they treat it like an array and leave the DB the heck alone.

For example, when displaying a bunch of users you want nice html when showing if they own any books , so you’ll probably end up checking user.books.count multiple times per user shown.  This will ask the db for a book count every time you call count and it will do it for all users shown.  Not fun.

Unfortunately, simply using size or any? instead will make no difference unless the relationship is already populated or coincidentally 0 is returned as, for some odd reason, this result is cached by Rails.

Additionally you should eager load if you’re accessing data within the relationship when rendering the user’s line  – you were doing this already ay?  If a ‘count’ of books is simply being used rather than book related data then still use size and any? as this may change later down the line and it’s no extra effort.

Now to throw a spanner in the works, named_scope is different, in fact it behaves the exact opposite. way to  size and any?.  If you call them on a named scope they grab the whole result set (ie all the models), tell you what you want to know and then discard the result set – EVERY TIME.

Named scopes are not like relationships as they don’t cache their results, so successive calls to a named scope will hit up the DB each time, ignoring any previous calls made to it.  Ironically, calling count on a named scope makes a ‘count(*)’ call to the DB so it’s much lighter weight than size or any?.

I almost thinking that this behavior is inconsistent and could be changed so that calling count, size and any? on an unpopulated relationship would cache the result for that request as it could actually change during execution potentially causing errors or show something weirdly.

This problem can also occur with caching as cached fragments are usually checked for in actions and then in the view.  The fragment could be expired by a different request in between these two checks causing the view to barf as the action won’t have set up any data thinking the view won’t need it.  I’ve had this unlikely situation happen multiple times and ended up building an in-request cache to combat it.

I feel a plugin, or even attempted patch to Rails welling up from within…

UPDATE: I forgot all about using length, which muddies the waters further!  Length populates the relationship and then returns it’s size.  Meaning subsequent calls don’t talk to the DB or if the relationship was already populated there isn’t any DB activity either – just like size and any?.  The big thing to note here though is that it attempts to populate the relationship first which you many not want happening.  Remember eager loading?

With named scopes it behaves the same, but since named_scopes behave differently to relationships there is no caching or populating happening – so each call involves the database returning all the records and Rails turning them into models.

I’m still going to be using size and any? as they’ll do ‘count(*)’ calls to the db if the relationship isn’t populated, otherwise they’ll just return it’s array size – or a boolean in the case of any?.  Meaning a git a bit of code future proof happening – ie less ‘why is that suddenly real slow?’ type situations.

No related posts.

I used to use relationship extensions lots and it always irked me that I was putting business logic for a model, say Book, on a related model, Owner, like this:

class Owner < ActiveRecord::Base
 
  has_many :books, :as => 'owner' do
    def old
      all(:conditions => ['published < ?', 20.years.ago])
    end
  end
 
end

Especially if multiple related models wanted the same logic:

class Company < ActiveRecord::Base
 
  has_many :books, :as => 'owner' do
    def old
      all(:conditions => ['published_at < ?', 20.years.ago])
    end
  end
 
end

So to eliminate any code duplication problems down the line I would put the business logic in a module within book.rb:

Module BookExtention
  def old
    all(:conditions => ['published_at < ?', 20.years.ago])
  end
end

and extend the relationships using that instead:

class Owner < ActiveRecord::Base
  has_many :books, :extend => BookExtention
end
 
class Company < ActiveRecord::Base
  has_many :books, :extend => BookExtention
end

That still didn’t completely resolve my discomfort though as the business logic was still outside the model itself. It was enough for me to move on to other problems though due to the code duplication issue being negated.

Why use extensions at all I expect you might think? It means in your actions you don’t have to continually repeat the condition and working through relationships means you have no security worries due to Rails taking care of that for you – walk the relationships for tighter more secure code people

Another way of getting a list of the user’s old books is this way:

current_user.books.select {|b| b.published_at < 20.years.ago }

Yep this works and is real quick on your dev machine with a limited dataset, say your fixtures for example. Here is what it’s doing. It’s getting ALL the user’s books from the DB, rails is then turning ALL that data into models, slow and memory hungry, and then the select is iterating over ALL the user’s books to find just the old ones. If you’re planning on people actually using your website, then use the DB for what it’s good at and filter the data as much as you can BEFORE it hands it over to your app.

Worse still is if this was done:

Books.all.select {|b| b.published_at < 20.years.ago }

Additionally, don’t use your fixtures for development data, use a recent backup form your production server, when that gets too big use a subset of that. Don’t have user generated data? Make some up. I jam chunks of Wikipedia into my database when I don’t have any real data to play with. Unless you do this you won’t know how your code performs in real life.

I’ve seen this style of coding quite a bit as Rails with it’s ORM wrapper, ActiveRecord, encourages developers to discard the shackles of SQL and do it all in ruby using rails with highly readable code. I’m all for that, but you have to write SQL as some point and this is where named scopes come in. They allow you to define (name) bunches of business logic (scope) and then string them together – which you can’t do with relationship extensions, for example:

Module BookExtention
 
  def old
    all(:conditions => ['published < ?', 20.years.ago])
  end
 
  def romantic
 
    all(:conditions => "genre = 'romantic'")
  end
 
end

What if I want old romantic books? I have to use one of the methods and then manually select based on the other’s condition. Ugh!

This is where named scope steps up to the plate and impresses the crowd, instead of the relationship extension do this in your model:

class Book < ActiveRecord::Base
 
  belongs_to :owner, :polymorphic => true
 
  named_scope :old, :conditions => ['published < ?', 20.years.ago]
  named_scope :romantic, :conditions => "genre = 'romantic'"
 
end

The beautiful thing here is that all your existing code which references the old methods off of the books relationship do not need to change as this still works:

current_user.books.old

Basically named_scope is jamming the :conditions into the current finder scope and applying that along with the relationships scope and defaulting to calling ‘all’. This means you can also do these:

current_user.books.old.all(:limit => 20)
Books.romantic.count

Instead of ‘all’ you can call what ever you normally could off of the relationship or the model’s class. And it generates the appropriate SQL given the scopes in play at the time. Nice. As the named_scope is just pushing itself onto the :find scope you can string them together like this:

current_user.books.old.romantic.all(:limit => 20)

This means that the DB will only return the data that you want, so no time or memory is wasted on data that would immediately be discarded by running a select over the users old books. Hurray.

As named_scopes are defined at class level the 20.years.ago is actually executed as the class definition is read into memory when the rails app first spins up. So if it was running for a year it would actually be 21.years.ago. To handle this you can do the following:

named_scope :old, lambda { {:conditions => ['published < ?', 20.years.ago]} }

Each time ‘old’ is used the lambda is executed returning a hash of finder options, in this case just a condition. Your eyes are probably lighting up at this stage thinking of the possibilities with dynamic named scopes and yes you can pass parameters to them. Thats for my next blog entry though sorry.

In summary, this gives two types of optimisation. Firstly higher quality though better readability, cleaner and tighter code, no code duplication and business logic living where it should be. Secondly, not only can you serve more requests from the same hardware due to faster execution, but because it’s more memory efficient it’s likely you can run more application instances, ie mongrel, on that same hardware.

So that covers the basics of why I like named_scope, in my next entry I’ll take this a couple of big steps further and possibly get into some nitty gritty log examinations.

No related posts.