Ultra Light & Maintainable Wizards in Rails

Recently I worked on Rails applications that had major requirements for web wizards, and they were built incrementally and iteratively with both junior and senior developers contributing further steps to the wizards over the lifetime of the projects. I presented my approach this past summer at the Montreal.rb user group (Update: and RailsConf 2014 ), and I am writing this blog post to provide more details and code examples.

Late last year and early this year when I scoured the net for web wizard approaches in Rails and wizard libraries, I noticed many different variations, happening across many dimensions, like the number of controllers/actions and how data is persisted across wizard pages.

Examples of controller/actions variations:

• One controller per wizard page
• One action per wizard page

Examples of Data persistence variations:

• Save data in the session on every page and create an ActiveRecord when finishing wizard
• Save data in a different ActiveRecord per wizard page
• Accumulate data in a hidden field and use on the last step to create an ActiveRecord
• Save data in one model with the help of a state machine
• Manage saving data in the controller

Some approaches try to be modular by having multiple controllers or multiple actions. Others try to be light on session use by saving to the database directly.

Here is my personal feedback on the approaches above:

• A wizard is editing one resource through multiple steps, so why are there multiple actions or controllers involved in building/updating that single resource? That violates REST. Not that it matters to break guidelines except for the pragmatic fact that it makes it more complex for developers and future team members to maintain the code.
• A single ActiveRecord is being edited across steps even if some steps edit sub-models within it, so saving different ActiveRecords is an over-complication. That violates Cohesiveness by fragmenting the focus of the data persistence of a wizard.
• Saving data to the session has server scalability concerns and complexity concerns. That makes Scalability harder.
• Accumulating data in a hidden field increases complexity. That goes against Simple Design.
• Managing data persistence details in the controller does not divide responsibilities correctly between the controller and model. That violates MVC, making it harder to maintain the code by developers and future team members.
• A Wizard is simply editing different facets (data views) of a model, and not necessarily moving a model along a Business State Machine. The steps for entering/editing a model are also a View/Presentation concern, so even if a state machine is involved, it is not a model concern. The model might have genuine business states, but are usually only few compared to the steps of the wizard, which might adjust for usability (presentation) reasons, so although there might be overlap between wizard steps and model business states, one must not mix them as that couples the two incorrectly and unnecessarily, complicating maintainability greatly (with much cost to pay by developers over a year). That violates MVC, abuses State Machine, and worsens Maintainability.

So, where does that leave us?

I ended up solving the problem on both Rails projects with a new simple and clean approach following some concrete goals.

Goal Summary

• Simplicity
• Maintainability
• Scalability
• Proper MVC
• Proper REST
• Proper Object Oriented Design (respect for coupling and cohesion principles)

Starting with this end in mind, one way to think about the wizard is that it is nothing but a model builder, akin of the good old Builder Design Pattern. Without a wizard, you usually have one form that a user fills in to create a model. With a wizard, the model is created in multiple steps however, so the REST resource edited in each step is a model part (aka model data view or model step).

Wizard Flow

• Starting the wizard creates the model and allows the user to edit the first part (data view) of the model
• Each subsequent step enables the user to edit more parts of the already created model

Guidelines

• REST: Following REST, the resource is nothing but the good old Model itself, except a wizard focuses on one part of the model at a time, so more accurately "model part" is the RESTful resource for wizards.
• MVC: Following MVC, each step can be routed to an ActiveRecord representing a part of the Model. To figure out which part, the Controller relies on the ID parameter, indicating the part (step) being edited in addition to model_id. Also, each model part is a subclass of the main Model ActiveRecord to facilitate connection to the database columns with the least code possible while retaining separation of concerns (an improvement would be to use delegation from a wrapper model, useful as a future refactoring step for the design if the coupling to ActiveRecord becomes undesirable later on. In my experience, that has not been an issue, so I elected to rely on single-table-inheritance style of database mapping reuse as you will see below)

With the two guidelines above, the developer has a complete template for the implementation with everything falling into place, and the rest is details. Let's get down and dirty to illustrate the concepts above with code!

Routes ProjectsController is only responsible for creating a project. Afterward, ProjectPartsController manages the project parts as wizard step resources (REST). The routes above will produce this pattern for moving along a wizard (editing a model part): /projects/:project_id/project_parts/:id/edit

ProjectsController
ProjectsController redirects to the first project part (wizard step) after project creation

ProjectPartsController
ProjectPartsController treats model parts (wizard steps) as the resource that can be edited/updated along the way. params[:id] contains the ID of the resource, which is the model part name (aka wizard step name). Rendered views can then match that ID name. The implementation above is one way to do it. The code could be written differently and optimized depending on needs (such as moving some private methods to a Helper and sharing them with the views if needed, like the step and next step for example, which can be displayed visually on the page for labelling and navigation)

Project model
The main project model contains associations and common logic, and then each wizard step can have its own sub-model if necessary to manage step specific validations without having the validations interfere with other steps. Sub-models also manage data loading upon showing the wizard step edit page (via after_initialize) or after updating (via after_save)

Project::BasicInfo sub-model


Project::Detail sub-model

Views

Note how the views match the wizard steps.

View Forms
All view forms (e.g. basic_info.html.erb) will have the form template above, which is agnostic to which model part is being edited. On one project, the developer actually created a helper (e.g. project_part_form_for) that hides all the details and can be unobtrusively reused across all wizard view pages.

Sub-Models

Note how only the wizard steps that require validation logic and special on edit or on update logic need sub-models.

I hope you found this example implementation helpful in your endeavour to implement Ultra Light & Maintainable Wizards in Rails. Note that it is a simpler version of what I have implemented for my clients, which often grew with their rising needs over time, stretching the implementation without much strain due to its simple design and adherence to sound software development guidelines, such as REST and MVC. This approach really scales well since it allows one to add extra authorization checks in the controller without a hitch and extra hooks/validations on the sub-models depending on need. Additionally, it is compatible with the model having a state machine or following the State Design Pattern as implemented on one of my projects.
Now, no write-up would be complete without a Gotchas section, so let's get to that.

Gotchas

• Validations live in sub-models. What if one wants to have a second mechanism for editing the model in one page utilizing the same validations? Simple. Chop the validations off into Modules, mix each of them into one of the wizard sub-models and then mix them all into a new sub-model to be used as the model for the second editing page. This approach can also be used to reuse validations for the last step of the wizard and re-run all the validations once again at the end if needed.
• How do you skip a step based on certain rules? You need to build a check into the controller that sees if a certain model part needs to be skipped (via a skip? method for example on the sub-model). If so, the controller simply redirects to the following step. Covering the details of this is outside the scope of the blog post, but I am sure you can figure it out.

On the first project that I added the wizard to, the CTO was happy to see that the junior developer was able to add new steps to the wizard in record time after spending 5 minutes with me going over the details of how it works. On that same project, a senior developer who joined later on and was working remotely figured out the wizard design in no time and sent me congratulations on how flexible and nicely designed it was. The same happened on the other Rails project I used the wizard approach for, which facilitated growing a wizard from about 5 steps to 15 (with some interesting custom logic on step branching). So, you can rest assured this approach will work well and scale for your Rails projects.

First Experience with Rails 4 Turbolinks

My most recently developed Rails application was built with Rails 4, which ships out of the box with a feature called Turbolinks. Here is its description from the Turbolinks github repo:

Turbolinks makes following links in your web application faster. Instead of letting the browser recompile the JavaScript and CSS between each page change, it keeps the current page instance alive and replaces only the body and the title in the head.

In effect, hyperlinks become turbolinks as page loading is perceived to be faster without the reloading of JavaScript and CSS despite having the same network load time for the webpage HTML markup.

In this post, I am sharing some of the gotchas I encountered with this feature, how I dealt with them, and my overall experience with the feature.

Gotchas and Solutions:

1. Style changes were not updating on page navigation

I encountered an issue the first time I tried to leverage the controller-specific CSS stylesheet pattern. For those unfamiliar with the pattern, it is actually about loading an extra CSS file in addition to application.css in order to customize the look of pages belonging to a specific controller. To enable the pattern, one would add a line like the following to their layouts/application.html.erb:

<%= stylesheet_link_tag params[:controller], media: "all" %>

This in effect, loads users.css for the page rendered by users#index or users#show for example.

The benefit of the pattern is simply keeping that CSS maintained separately from other pages so that when the Rails Asset Pipeline builds application.css it does not concatenate many potentially conflicting styles and forcing developers to deal with them with all sorts of complicated hacks. Having a controller-specific CSS loaded per controller page in addition to application.css results in much lighter more maintainable page specific styles.

So, what was the gotcha?

Well, upon updating the controller specific CSS, I noticed that its styles were not reflected when I was navigating between the controller pages until I refreshed the browser page completely. Turns out, that is because I did not have turbolinks tracking enabled, so I had to add this option to resolve the problem: , "data-turbolinks-track" => true, changing the erb to:

<%= stylesheet_link_tag params[:controller], media: "all", "data-turbolinks-track" => true %>

2. On-document-ready JavaScript events were not firing on page navigation

For example, if you have JavaScript code that converts form hints into tooltips like the following code (in CoffeeScript):

$ =>
  $('.hint').tooltip(placement: 'right')

The code above relies on jQuery's shortcut method $ for executing some JavaScript (CoffeeScript) code upon page document load (document ready).

When visiting the form for the first time, the tooltips show up. However, when navigating away to the home page, and navigating back to the form page, the tooltips do not show up anymore.

Turns out that it is expected behavior by Turbolinks' definition. After all, the Turbolinks feature does not reload JavaScript files on page navigation, missing out on the activation of hint tooltips on document load.

Fortunately, the Turbolinks feature offers developers a way to get around this through its own event hooks fired on page navigation. They are detailed in the Turbolinks github repo (highly recommended to visit to learn Turbolinks in depth).

The event hook I used in particular was: "page:load" (2020 update: "turbolinks:load"). Using it alone did not resolve the problem however as it worked for page navigation, but not initial page load (2020 update: the docs claim the new "turbolinks:load" now works for both initial page load and later page visits), so I used it in addition to the original jQuery $ method as follows:

$(document).on "page:load", =>

  $('.hint').tooltip(placement: 'right')

So in essence, the gotcha fix was to wrap all the page document load logic in a JavaScript function, and then hook it into both on-document-ready (jQuery $) (2020 update: the docs claim this is no longer needed with "turbolinks:load") and Turbolinks "page:load" (2020 update: now "turbolinks:load")

Experience:

Now regarding my overall experience with Turbolinks, I thought it was a very welcome addition to Rails. It worked quite perfectly once I have gone beyond the two gotchas, resulting in an impressively instant page navigation and making local browser testing much faster and more productive. The JS libraries I have used with Turbolinks by the way are the Spectrum Color Picker, Twitter Bootstrap Tooltips, and Fancybox in addition to usage of JS media queries.

In a way, Turbolinks is the perfect embodiment of the Rails philosophy of discovering common patterns, automating their solutions to be as effortless and friction free as possible for everyone to use, and finally offering the ability to customize or disable all-together. By the way, you can opt out of Turbolinks by adding the attribute "data-no-turbolink" to the HTML body element or divs for which you want links to behave regularly. In essence, Turbolinks' implementation satisfies the 80/20 rule, automating the solution for 80% of the cases (not literally) to increase productivity and allowing developers to customize the solution in the remaining special 20% of the cases (not literally) to allow flexibility. Using Turbolinks is also an instance of following Convention over Configuration.

In conclusion, Turbolinks is like driving your first Porsche. The first couple of times you drive the car, you are out of control and do not know why the hell the car is not behaving like a good old Honda, but then you finally get the hang of it, and suddenly find yourself flying in a 911 Turbo. ;)

Managing Rails Routes When They Get Out Of Hand

Rails developers have gotten really good over the years in breaking down logic in beefy controllers to business domain logic that lives in Models and some reusable control logic that lives in Controller Modules. Not only does this address cohesion concerns, but also the lines per file recommended limit of no more than say 200 in Ruby or 400 max.

However, what I have not seen done often enough in most of the projects I've worked on is handling of these concerns in the Rails Routes file "routes.rb". I've often seen gigantic routes files that not only take a long time to detangle and understand, but also aren't organized in any fashion as to the grouping of routes per business domain area (e.g. Feature A routes, Feature B routes, etc...), which brings us to the topic of this blog post. :)

If you simply follow the software engineering recommendation to break files into smaller ones once they've surpassed a limit of say 400 lines, then a route file (typically 500+ on bigger Rails projects and sometimes 1000+ or even 2000+) is naturally to be broken into multiple route files.

There are several techniques for doing it, but here is one easy technique that I've used on the last couple of Rails projects I was on:

1. Break routes.rb along these lines: Static Routes, Admin Routes, Account Routes (e.g. authentication with devise), and Per Feature Routes (e.g. Company Management Routes, User Personalized Info Routes, etc...)

2. Create directory "config/routes" to store smaller route files

3. Create a route file under "config/routes" for each one of the functional areas mentioned in Step 1, named "XYZRoutes" and holding a Ruby module like the following example:

module Routes

  module AccountRoutes

    def self.draw(context)

      context.instance_eval do

        devise_for :users, :controllers => { :registrations => "registrations", :sessions => 'sessions' }

        devise_scope :user do

          get "/login" => "sessions#new"

          delete '/logout' => 'sessions#destroy'

          get '/logout' => 'sessions#destroy'

          get '/register' => 'registrations#new'

        end

      end

    end

  end

end

4. Update routes.rb to look like this:


Dir.glob("#{Rails.root.to_s}/config/routes/**/*.rb").each {|route_file| load(route_file)}

SomeApp::Application.routes.draw do
  Routes::StaticRoutes.draw(self)
  Routes::AdminRoutes.draw(self)
  Routes::CompanyRoutes.draw(self)
  Routes::AccountRoutes.draw(self)
  Routes::UserRoutes.draw(self)

  mount JasmineRails::Engine => "/specs" if defined?(JasmineRails)

  root :to => "home#index"
end

This should provide you with a good blueprint for how to better organize routes files in Rails.

To summarize, break down routes.rb in Rails when it gets over 400 lines of code for the following benefits:

• Higher cohesion per route file, improving general understanding of that area's focus as well as ease of maintainability by allowing you to find URLs for a particular domain area faster.
• Better management of route content by not having to scroll through 100s of lines of code (sometimes 1000s) to find the URL that you want to change

I've worked in an environment once where the development team wrote an internal library to streamline modularization of routes.rb. If you know of a public open source one, please mention in comments.

2022-03-01 Update:

Rails now supports breaking up route files natively through the `draw` macro:

https://guides.rubyonrails.org/routing.html#breaking-up-very-large-route-file-into-multiple-small-ones

# config/routes.rb

Rails.application.routes.draw do

  get 'foo', to: 'foo#bar'

  draw(:admin) # Will load another route file located in `config/routes/admin.rb`

end

# config/routes/admin.rb

namespace :admin do

  resources :comments

end

This should have been mentioned a while ago, but better late than never I guess.

Rails 3.1 Asset Syncing to S3 on Heroku

Rails 3.1's Asset Pipeline is a wonder in management of static assets in modern web development. It streamlines the process of gzipping, minifying, and fingerprinting asset files as well as setting HTTP cache headers and integrating into a reverse-proxy cache server such as Rack-Cache (comes with Rails) or Varnish.

On Heroku's new Cedar stack however, one cannot use a high-performance static asset server like Nginx or even a high-performance reverse-proxy server like Varnish. So, the next high-performance option, which is arguably higher-performance is remote asset hosting on Amazon S3 or a CDN (Content Distribution Network) like Amazon Cloudfront.

Fortunately, one gem - asset_sync - comes to the rescue. It automates syncing of static assets from your app to S3 or a CDN upon asset compilation.

I used it to serve static content from S3 for a website I am architecting, so I would like to share these tips with anyone who has attempted to follow Heroku's asset syncing guide:
https://devcenter.heroku.com/articles/cdn-asset-host-rails31

If you try to have it sync automatically on deploy to Heroku, it fails miserably due to Heroku not loading its ENV variables in production mode. We missed that when we deployed to staging because asset syncing was working there during the Heroku asset compilation step.

To avoid falling for that trap, you need to compile assets locally, commit, and push. Unfortunately, this can get painful with setting all the ENV vars for asset_sync or passing it options that are supposed to be secure passwords you do not want to put in the git repo.

I wrote a rake task (inspired by a colleague's rake file) to automate the process of grabbing the ENV vars from Heroku instead, compiling, committing, pushing, and then deploying to heroku all in one command.

Code:
https://gist.github.com/4273735

Usage:
rake deploy app=heroku_app_name branch=branch_to_deploy

Alternate syntax:
rake deploy[heroku_app_name,branch_to_deploy]

Default branch as master:
rake deploy[heroku_app_name]

Heroku DB Migration from PostgreSQL to ClearDB

I have been recently working as the software architect of a Ruby on Rails application running on Heroku, and as part of the work, we decided to migrate off of Heroku's default PostgreSQL database into the MySQL based ClearDB.

To do so, I tried relying on Heroku's taps gem, but it produced mediocre results with some MySQL database columns not matching the same null or default value constraints in the source PostgreSQL database.

I ended up coming up with my own approach to perform a successful perfect database migration, and I would like to list the steps over here for anybody else who might undertake the same endeavor since the Heroku and ClearDB guides do not include any migration instructions as of today from the Heroku PostgreSQL database.

Prerequisites

• Setup PostgreSQL database ("brew install postgresql" then follow post-installation instructions. version 9 would do)
• Setup ClearDB on your Heroku app and add SSL keys to your application as per their instructions
• Obtain pg2mysql_cli.php tool from over here: http://www.lightbox.ca/pg2mysql.php and extract it to a local directory, under which you will run the commands mentioned under the instructions section.
• Obtain the URLs for the PostgreSQL and ClearDB databases from Heroku by running the command "heroku config -a appname"
• Extract the username, password, host, and database name from the database URLs as mentioned in the Heroku PostgreSQL guide and ClearDB guide
• For example: 'mysql2://someuser:somepassword@us-cdbr-east-02.cleardb.com/somedatabase?reconnect=true' should yield: username "someuser" password "somepassword" host "us-cdbr-east-02.cleardb.com" and database "somedatabase"

Migration Instructions

Turn on Heroku's maintenance page to stop write traffic into the database

heroku maintenance:on -a appname

Backup the Heroku PostgreSQL database

heroku pgbackups:capture -a appname

Download the Heroku PostgreSQL backup

curl -o b001.dump `heroku pgbackups:url -a appname`

Restore PostgreSQL backup locally in order to convert to uncompressed format

pg_restore --verbose --clean --no-acl --no-owner -h localhost -U local_postgres_db_user -d postgresdbname b001.dump

Generate new uncompressed data-only backup (format most appropriate for ClearDB)

pg_dump -U local_postgres_db_user --data-only --column-inserts --format p -f b001-data.sql postgresdbname

Run pg2mysql_cli.php command from the pg2mysql tool directory to convert PostgreSQL backup to MySQL backup

php pg2mysql_cli.php b001-data.sql b001-data-mysql.sql

Clear ClearDB MySQL DB completely

MYSQL="mysql -h us-cdbr-east-02.cleardb.com -u someuser -psomepassword -D somedatabase"
 $MYSQL -BNe "show tables" | awk '{print "set foreign_key_checks=0; drop table `" $1 "`;"}' | $MYSQL
 unset MYSQL

Set ClearDB as the main DB for the Heroku app (assuming you are relying on SSL for security and followed the ClearDB instructions for settings it up)

heroku config:add DATABASE_URL='mysql2://someuser:somepassword@us-cdbr-east-02.cleardb.com/somedatabase?sslca=cleardb-ca.pem&sslcert=someuser-cert.pem&sslkey=someuser-key.pem&reconnect=true' -a appname

Initialize ClearDB schema from schema.rb in the Rails app

heroku run 'rake db:schema:load' -a appname

Clear any database tables that got populated in the schema initialization process

MYSQL="mysql -h us-cdbr-east-02.cleardb.com -u someuser -psomepassword -D somedatabase"
 $MYSQL -BNe "show tables" | awk '{print "set foreign_key_checks=0; delete from `" $1 "`;"}' | $MYSQL
 unset MYSQL

Load ClearDB with MySQL backup data obtained from the Heroku PostgreSQL DB

mysql -u someuser -psomepassword -h us-cdbr-east-02.cleardb.com -D somedatabase < b001-data-mysql.sql

Migrate

heroku run 'rake db:migrate' -a appname

Seed

heroku run 'rake db:seed' -a appname

Turn off Heroku's maintenance page

heroku maintenance:off -a appname

That's all folks! Questions and comments are welcome.

Software Craftsmanship vs Software Engineering at McHenry Software Craftsmanship Meetup

Here are the slides of the long edition of my recent talk "Software Craftsmanship vs Software Engineering", which was given at McHenry Software Crafstmanship, Chicago Software Craftsmanship, and South Bend Software Craftsmanship:

Software Craftsmanship VS Software Engineering

View more PowerPoint from AndyMaleh

RailsConf 2012 - My Talk Slides

Here are the slides to my talks at RailsConf 2012.

Software Craftsmanship vs Software Engineering (Lightning Talk)

Rails Engine Patterns

It was nice meeting all of you at the conference. For those looking for a software development opportunity with self-growth potential, hit me up by email (last slide). Cheers!