Google for how to do it and you’ll quickly encounter basically this solution where you symlink in the system folder to trigger it:

  if ( -f $document_root/system/maintenance.html ) {
    rewrite  ^(.*)$  /system/maintenance.html last;
    break;
   }

Great, except that maintenance pages ofter have images in them too, and this rewrites those image requests to returning the maintenance page html. You’d think you could do a multi-conditions if statement and have something like this which only rewrites when the request isn’t for stuff under the system folder:

  if ( -f $document_root/system/maintenance.html && !$uri ~ ^/system/ ) {
    rewrite  ^(.*)$  /system/maintenance.html last;
    break;
   }

That would be sensible, but nope doesn’t work. So a bit more googling throws up this ruby-forum post which initially solved the problem nicely for me using a local variable in the config:

  if ( -f $document_root/system/maintenance.html ) {
    set $maintenance 1;
  }

  if ( $uri ~ ^/system/ ) {
    set $maintenance 0;
  }

  if ( $maintenance ) {
    rewrite  ^(.*)$  /system/maintenance.html last;
  }

Even though I was left with the sense of having just used a kludge at the time, I happily moved onto the next yak to shave. That was until sometime later when I moved Ponoko to Nginx. In the intervening time local variables in Nginx as I knew them had stopped working in that once it was set you couldn’t change it to something else. Back to square one.

A bit of thinking and I hit upon a much more elegant solution – host the maintenance page assets on a sub-domain which would be controlled by a different Nginx server config and not have any rewrite problems:

  server {

    listen 80;
    server_name maintenance.somedomain.com;

    # doc root
    root ;

    location / {

      if (-f $request_filename) {
        break;
      }

    }

  }

I then edited our maintenance page to use the new subdomain for it’s images etc… and changed the main server conf back to using the original suggestion:

  if ( -f $document_root/system/maintenance.html ) {
    rewrite  ^(.*)$  /system/maintenance.html last;
    break;
   }

Problem solved, and it stopped feeling like a kludge.

No related posts.

Last year I started a personal project to up-skill in some aspects of web development I was lacking in, like Javascript.  I had an idea for a game I’d been bouncing around for several years so I created that.  One thing led to another and now it’s become quite a serious project.

The main page for ESDAO gives an overview of what it is about, you can use this invite link to take the game for a spin.

There is a small base of committed testers enjoying the game – but it’s not really growing much and I’m kinda stumped about the best way to fix that.  I have some ideas but I have little experience in this type of thing and would love to get advice from someone with a successful background doing just that.

The game itself is close to completion and my next major tasks are getting more people playing it and sorting out the look and feel.  I have a designer lined up, but have yet to commission a site design while I work out what to do next.

So, I’m after an advisor who I can meet on a regular basis.  Someone who I can pay for good objective advice.  I’m based in Wellington and would obviously prefer someone local.

If you’d like to know more about me I have a page for that.

I’m best contacted via email – nahum.wild@gmail.com

No related posts.

The plugin is still useful as I didn’t submit it’s session caching changes to Rails – essentially using Rails.cache for session storage rather than establishing a second connection to MemCache.  This also benefits from the local cache changes.

Where next?  In the next week or so I plan on updating the plugin by removing the local cache functionality to prevent any conflicts with Rails 2.3 when it’s released.

After that, I’m planning on working through the list of changes detailed under Future Plans in the plugins readme, also listed below.

• TESTS! (I wrote a bunch for the patch, just need to do that for the remaining session changes)
• Make this a Gem
• Conditional caching
• Multi-Get
• Dynamic key helpers
• Make the code more awesome

No related posts.

I received two suggestions for the ‘best install method’, firstly MacPorts and then later pre-compiled dmg’s from Kyng Chaos.  I already use ports extensively making that choice easy for me – I like being able to  uninstall things if they don’t work to well and was concerned the dmg option wouldn’t allow me to.

sudo port install postgresql83 postgresql83-doc postgresql83-server postgis

I then added the following to my path in ~/.zshrc so I could actually run things

/opt/local/lib/postgresql83/bin

and started the server.

sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql83-server.plist

All seemed to be fine and dandy as I could connect with pgAdmin.  I quickly discovered that I needed to create a user with the same login as my osx login when I tried creating a test db.

Next I tried restoring my client’s db dump and it barfed big time.  As I’ll be working on some mapping software I’d installed the PostGIS package which seems to have failed to link up with Postgres properly.   I was getting the follow missing library error:

could not load library /usr/lib/postgresl/liblwgeom.so.1.3

Obviously it’s looking in the wrong place for a start, so I setup a LIBDIR export in my profile to get over that problem and restarted the server.

export LIBDIR=/opt/local/lib/postgresql83

That still didn’t solve the problem though as I don’t have a file by that name on my system, I do however have liblwgeom.1.3.so which looked to be the same thing.  Pretty annoying, but easily solved with a symlink.

cd /opt/local/lib/postgresql83
sudo ln -snf liblwgeom.1.3.so liblwgeom.so.1.3

Then the backup restored like a chalm.

To get Rails talking to Postgresql I needed to install the postgres adapter gem appropriately called ‘postgres’.  I didn’t expect that to work straight off though as I’d installed postgresql with ports and the gem by default would be looking in the wrong place for it.  I was correct, but I also received an additional error complaining about the architechture not being specified – I needed to tell it I wanted i386.

sudo env ARCHFLAGS="-arch i386" gem install postgres -- --with-pgsql-include-dir=/opt/local/include/postgresql83l --with-pgsql-lib-dir=/opt/local/lib/postgresql83/

I gropped around for a bit trying to figure out the correct way of letting it know and eventually found the above solution on code snippets.

No related posts.

You’ve noticed that your app is doing a lot of repetitive work – calculating and showing the same data over and over again.  This doesn’t fit with your adherence to DRY principles at all.  If you cached repetitive responses your servers could give you way more bang for you buck – and it would only cost your time.

You are about to embark on an exciting journey fraught with hidden dangers, so let me prepare you – cheesy I know but hey.

First, you have actually done some code optimisation haven’t you?  Stopped using the default mysql config?  Checked your slow query log?  Used pl_analyise to parse your production logs?  Fixed the popular, but slow actions?  Done some eager loading so you get everything in a couple of hits from mysql rather than many many many little hits as your views render?  When you dream you see log tailing in the background?  Minified your javascript and css?  Static assets have expire headers?  Don’t have too many Mongrels running?  Sessions are stored in MemCache? Go do all that if you haven’t and only come back here if this are still too slow.  Oh and try to get your yslow score above 75.

Note: I’m assuming that you have read a bit about caching in Rails already and have had a play, reading fragments from actions etc…  If not try googling as there are a number of great articles on it, the Agile book is probably a good start too.

Start at the end

Instead of just saying ‘do this awesome thing’ I’m gonna explain how I’ve gotten to saying that, as you shouldn’t be making blind decisions about these things.  Most of the time when I implement caching for clients I completely skip page caching and action caching and go straight to line item caching, ie caching of small fragments of html.  Sounds completely counter-intuitive doesn’t it, but it’s not.

First off I need to define a typical client’s website so you see where I’m coming from.  It’s a user driven web 2.0 website – so most of the data is user generated and sometimes social networking elements are included, where a user’s data can be restricted to just their friends instead of being publicly available.  Basically the site lists stuff that people put on it.

Why not page or action caching?  As a starting point it’s too hard to manage due to the continually changing nature of the data being presented.  This changing nature means your cache might not actually get used that much.  Additionally when you are logged in your name is typically on the page with other unique to you info – you have x messages etc…  The latter reason completely rules out caching a whole page for anyone that’s logged in.  Don’t restrict caching based speed increases to joe public as logged in people, ie people that give a damn about your site, shouldn’t be second class citizens.

Don’t start with caching lists

Developers still want to jump straight in and cache lists of items (and pages too) rather than the individual line items – basically as much of the page as they can in a single hit – thinking they’ll get great bang for their buck.  When you deploy with that strategy you’re in for a world of hurt.  This is because they’ve not realised they need to take the following into account.

Each time one of the line items changes you have to expire the cache for the whole listing.  If you have pagination then you’ll likely have to expire ALL cached pages unless you can 100% reliably work out which page the item is on.  So yes you have a separate cache entry for each page of the listing.  If someone adds or edits a line item you have to expire the whole lot – and thats what people should be doing all the time on your site right?.  Additionally, if you allow sorting or a filtering mechanism you have to uniquely cache by page and the additional dimension(s), thus further reducing the frequency the cache will be used.

What’s the goal?

What we are trying to achieve with caching is to DRY up the execution of requests, but in a way that doesn’t add complexity.  The idea is to let your servers focus on the unique stuff like processing credit cards and telling Jill’s friends that based on her network and comment history she really doesn’t like doing things with Sarah unless Joe is there too.

Keep this in mind.  With seeking something to cache, you are basically looking for something that has a high ratio of renders/requests of the exact same html in between changes of that html.  So the more content in the cached fragment, the more frequently you’ll need to expire it.  Putting that another way, the fewer data sources, instances of models and display dimensions like pagination, sorting and filtering used to make up the cached html the better.  The simplest you can go is one data source, ie a line item.  Remember, keep it simple.

Put in business speak, it’s a cost / benefit decision.

It’s all about the keys

When you store fragments in your cache you name them with a unique string based key so you can access them later -  exactly like a hash.  So yes, they are important.  Typically they looks like a url, sort of based on the actual url or controller/action and the dimensions involved to make it unique to the html being cached.  So for the above listing situation the key should look something like this:

‘/products/list/sort=price/status=in-stock/page=1′

This actually presents a completely separate reason as to why you shouldn’t do list caching when you start out on this path.  The problem with this key is that to expire it you have to count the number of products to determine how many pages there are and then manually go and expire all possible keys that could have been generated, one at a time – this is because expiring via regex is not supported.  If additional dimensions are involved the problem will compound even further.

Yes, that’s real ugly and will become completely unmanageable.  There are some clever tricks to help out there such as self expiring keys, but that’s a different topic which I’ll get to at a later date.

When to expire

Wait there’s more!  You have to monitor data changes so you can expire those keys when they become stale as it doesn’t happen magically.  Normally you’d setup a sweeper (observer) and just expire things when ‘after_save’ is triggered.  Not a problem.  If you have data from multiple models involved in your cached html fragments then you need to monitor all the models involved.  You can go as far as only expiring particular keys based on the attributes that are changing etc… now much easier in Rails 2.1 with Dirty Objects.

This is all adding yet another layer of complexity.  Ugh.  It should all be documented so that other developers know about it especially new developers – otherwise further work could mess it up and you might start showing stale data to users.  In my book that’s pretty much on par with errors occurring as you could be showing the wrong price or even deleted data to users.  Unless you are careful about it, it can all become some really hard to manage mess which really isn’t going to give much benefit to your users.

Line item caching

As you are not having to worry about other dimensions such as pagination, sort order and filtering, other line items or even if the user is logged in or not, the line item’s cache key should be something like this:

‘/products/#{product.id}/line_item’

To make sure it expires correctly all you have to do is setup a sweeper watching the line item and simply delete it’s key when it changes – use Rails.cache.delete for this.  If the line item is rendered using data from through relationships, you then have to monitor them too.  I’ve seen people putting a ‘last_active_at’ field on a line item and when anything related to it changes they update that time field.  That works nicely and things are kept simple.

Finally

That should set you off in the right direction, remembering though this is a starting point for your first caching strategy.  Later on you’ll actually start doing multiple layers of caching and covering more complex html, such as lists, if your site continues to grow.

Now hopefully I’ve convinced you to start out with caching the html for a single line item – or something just as simple.  If not post a comment and tell me why not.

No related posts.

For me the reasoning is distilled down to my programming style.

Pragmatism

When it comes down to it, what vital benefit does it give you over :dependent => :destroy or model validation?  Will this give you increased quality faster, quicker and cheaper?

Premature Optmisation

Basically don’t waste time optimising something that isn’t slow or is hardly used.  Rails is so slow that some fractional speedup on the DB isn’t really going to be noticeable by the end user.  Instead examine your site with NewRelic or FiveRuns and fix your bottleneck code – way more bang for your buck.

Data is Valuable

So don’t go around deleting it!  If a user wants to delete some data you should want to know why.  Looking at the data can aid in determining why without having to bug the user about it.  Yes there are occasions when this does need to happen, but they should be the exception to the rule and thus explicitly coded for.

Just mark a record as deleted and filter it out automatically, that’s easy enough to do with rails – way easier than any other language I’ve used before.  Better yet, search github and use someone else’s wheel.

Convention Over Configuration

Basically put, you do things the way that everyone else has done it.  Walk the well walked path so to speak’ and you won’t experience any bizzaro problems that someone else hasn’t already solved before you.  As you’ve created your app in a predictable way, someone else can easily come in and contribute to your project, or visa-versa and it makes the whole Rails community more accessible to you.

So in following the conventions and as you are not encouraged to use foreign key constraints, Rails isn’t expecting data to be deleted out from underneath it.

Don’t Repeat Yourself && Model View Controller

DRY is another of the core philosophies behind Rails – partials are a real good example of this especially when you start to think of them as essentially methods.  On top of the obvious ’speed to deliver’ benefits this concept provides, it also causes you to distill your code down to being more readable and accessible.  This ties in nicely with MVC principles as you then tend to group like logic together which is what MVC is all about.

Foreign key constraints are business logic as they describe and implement a behavior – “when you delete this record, also delete these other records over here”.  So why would you go and put some ‘important to know about’ business logic in an isolated and often hard to reach place?

Documentation

The one great thing I learned from Steve McConnell’s Code Complete was how to write comments, they should describe the intention of the code, rather then repeat the code.  This nicely ties in with the DRY principle.

I’m also a great believer in self documenting code, Rails and Ruby help heaps with this, and generally write comments to help simplify complex code.  Instead of my code being littered with redundant comments cluttering everything up, it just contains important ones as they are more likely to be read and updated when the code is next worked on.

If you use foreign key constraints in your DB you’ll likely end up documenting them (ie repeating them) somewhere more accessible, in the models being an obvious place.  That wouldn’t be because the business logic lives there would it?.  Have a think about that.

The Rails Way

Looking at the Convention over configuration, Don’t Repeat Yourself and Model View Controller concepts above – as much as Rails is a framework it’s also about a way of doing things.

Over the years I’ve seen a notable number of people, new to rails and these concepts, trying to make rails work their way – often with less than desirable results.  If you embrace Rails and it’s ‘way’ then it works beautifully and is a pure joy to use, rather than a frustrating experience.

For me

Coding is an art form and as such a creative process.  If something doesn’t look, or even feel right then it’s wrong.  In the end foreign key constraints on a web app just rub me up the wrong way.

No related posts.

Put basically if you call count on a relationship it will always make a db call to see how many there are even if the relationship has been populated.  If you use size or any? they will too, unless the relationship is populated, in which case they treat it like an array and leave the DB the heck alone.

For example, when displaying a bunch of users you want nice html when showing if they own any books , so you’ll probably end up checking user.books.count multiple times per user shown.  This will ask the db for a book count every time you call count and it will do it for all users shown.  Not fun.

Unfortunately, simply using size or any? instead will make no difference unless the relationship is already populated or coincidentally 0 is returned as, for some odd reason, this result is cached by Rails.

Additionally you should eager load if you’re accessing data within the relationship when rendering the user’s line  – you were doing this already ay?  If a ‘count’ of books is simply being used rather than book related data then still use size and any? as this may change later down the line and it’s no extra effort.

Now to throw a spanner in the works, named_scope is different, in fact it behaves the exact opposite. way to  size and any?.  If you call them on a named scope they grab the whole result set (ie all the models), tell you what you want to know and then discard the result set – EVERY TIME.

Named scopes are not like relationships as they don’t cache their results, so successive calls to a named scope will hit up the DB each time, ignoring any previous calls made to it.  Ironically, calling count on a named scope makes a ‘count(*)’ call to the DB so it’s much lighter weight than size or any?.

I almost thinking that this behavior is inconsistent and could be changed so that calling count, size and any? on an unpopulated relationship would cache the result for that request as it could actually change during execution potentially causing errors or show something weirdly.

This problem can also occur with caching as cached fragments are usually checked for in actions and then in the view.  The fragment could be expired by a different request in between these two checks causing the view to barf as the action won’t have set up any data thinking the view won’t need it.  I’ve had this unlikely situation happen multiple times and ended up building an in-request cache to combat it.

I feel a plugin, or even attempted patch to Rails welling up from within…

UPDATE: I forgot all about using length, which muddies the waters further!  Length populates the relationship and then returns it’s size.  Meaning subsequent calls don’t talk to the DB or if the relationship was already populated there isn’t any DB activity either – just like size and any?.  The big thing to note here though is that it attempts to populate the relationship first which you many not want happening.  Remember eager loading?

With named scopes it behaves the same, but since named_scopes behave differently to relationships there is no caching or populating happening – so each call involves the database returning all the records and Rails turning them into models.

I’m still going to be using size and any? as they’ll do ‘count(*)’ calls to the db if the relationship isn’t populated, otherwise they’ll just return it’s array size – or a boolean in the case of any?.  Meaning a git a bit of code future proof happening – ie less ‘why is that suddenly real slow?’ type situations.

No related posts.