Rails 3.1 Asset Syncing to S3 on Heroku

Rails 3.1's Asset Pipeline is a wonder in management of static assets in modern web development. It streamlines the process of gzipping, minifying, and fingerprinting asset files as well as setting HTTP cache headers and integrating into a reverse-proxy cache server such as Rack-Cache (comes with Rails) or Varnish.

On Heroku's new Cedar stack however, one cannot use a high-performance static asset server like Nginx or even a high-performance reverse-proxy server like Varnish. So, the next high-performance option, which is arguably higher-performance is remote asset hosting on Amazon S3 or a CDN (Content Distribution Network) like Amazon Cloudfront.

Fortunately, one gem - asset_sync - comes to the rescue. It automates syncing of static assets from your app to S3 or a CDN upon asset compilation.

I used it to serve static content from S3 for a website I am architecting, so I would like to share these tips with anyone who has attempted to follow Heroku's asset syncing guide:
https://devcenter.heroku.com/articles/cdn-asset-host-rails31

If you try to have it sync automatically on deploy to Heroku, it fails miserably due to Heroku not loading its ENV variables in production mode. We missed that when we deployed to staging because asset syncing was working there during the Heroku asset compilation step.

To avoid falling for that trap, you need to compile assets locally, commit, and push. Unfortunately, this can get painful with setting all the ENV vars for asset_sync or passing it options that are supposed to be secure passwords you do not want to put in the git repo.

I wrote a rake task (inspired by a colleague's rake file) to automate the process of grabbing the ENV vars from Heroku instead, compiling, committing, pushing, and then deploying to heroku all in one command.

Code:
https://gist.github.com/4273735

Usage:
rake deploy app=heroku_app_name branch=branch_to_deploy

Alternate syntax:
rake deploy[heroku_app_name,branch_to_deploy]

Default branch as master:
rake deploy[heroku_app_name]

Heroku DB Migration from PostgreSQL to ClearDB

I have been recently working as the software architect of a Ruby on Rails application running on Heroku, and as part of the work, we decided to migrate off of Heroku's default PostgreSQL database into the MySQL based ClearDB.

To do so, I tried relying on Heroku's taps gem, but it produced mediocre results with some MySQL database columns not matching the same null or default value constraints in the source PostgreSQL database.

I ended up coming up with my own approach to perform a successful perfect database migration, and I would like to list the steps over here for anybody else who might undertake the same endeavor since the Heroku and ClearDB guides do not include any migration instructions as of today from the Heroku PostgreSQL database.

Prerequisites

• Setup PostgreSQL database ("brew install postgresql" then follow post-installation instructions. version 9 would do)
• Setup ClearDB on your Heroku app and add SSL keys to your application as per their instructions
• Obtain pg2mysql_cli.php tool from over here: http://www.lightbox.ca/pg2mysql.php and extract it to a local directory, under which you will run the commands mentioned under the instructions section.
• Obtain the URLs for the PostgreSQL and ClearDB databases from Heroku by running the command "heroku config -a appname"
• Extract the username, password, host, and database name from the database URLs as mentioned in the Heroku PostgreSQL guide and ClearDB guide
• For example: 'mysql2://someuser:somepassword@us-cdbr-east-02.cleardb.com/somedatabase?reconnect=true' should yield: username "someuser" password "somepassword" host "us-cdbr-east-02.cleardb.com" and database "somedatabase"

Migration Instructions

Turn on Heroku's maintenance page to stop write traffic into the database

heroku maintenance:on -a appname

Backup the Heroku PostgreSQL database

heroku pgbackups:capture -a appname

Download the Heroku PostgreSQL backup

curl -o b001.dump `heroku pgbackups:url -a appname`

Restore PostgreSQL backup locally in order to convert to uncompressed format

pg_restore --verbose --clean --no-acl --no-owner -h localhost -U local_postgres_db_user -d postgresdbname b001.dump

Generate new uncompressed data-only backup (format most appropriate for ClearDB)

pg_dump -U local_postgres_db_user --data-only --column-inserts --format p -f b001-data.sql postgresdbname

Run pg2mysql_cli.php command from the pg2mysql tool directory to convert PostgreSQL backup to MySQL backup

php pg2mysql_cli.php b001-data.sql b001-data-mysql.sql

Clear ClearDB MySQL DB completely

MYSQL="mysql -h us-cdbr-east-02.cleardb.com -u someuser -psomepassword -D somedatabase"
 $MYSQL -BNe "show tables" | awk '{print "set foreign_key_checks=0; drop table `" $1 "`;"}' | $MYSQL
 unset MYSQL

Set ClearDB as the main DB for the Heroku app (assuming you are relying on SSL for security and followed the ClearDB instructions for settings it up)

heroku config:add DATABASE_URL='mysql2://someuser:somepassword@us-cdbr-east-02.cleardb.com/somedatabase?sslca=cleardb-ca.pem&sslcert=someuser-cert.pem&sslkey=someuser-key.pem&reconnect=true' -a appname

Initialize ClearDB schema from schema.rb in the Rails app

heroku run 'rake db:schema:load' -a appname

Clear any database tables that got populated in the schema initialization process

MYSQL="mysql -h us-cdbr-east-02.cleardb.com -u someuser -psomepassword -D somedatabase"
 $MYSQL -BNe "show tables" | awk '{print "set foreign_key_checks=0; delete from `" $1 "`;"}' | $MYSQL
 unset MYSQL

Load ClearDB with MySQL backup data obtained from the Heroku PostgreSQL DB

mysql -u someuser -psomepassword -h us-cdbr-east-02.cleardb.com -D somedatabase < b001-data-mysql.sql

Migrate

heroku run 'rake db:migrate' -a appname

Seed

heroku run 'rake db:seed' -a appname

Turn off Heroku's maintenance page

heroku maintenance:off -a appname

That's all folks! Questions and comments are welcome.

Software Craftsmanship vs Software Engineering at McHenry Software Craftsmanship Meetup

Here are the slides of the long edition of my recent talk "Software Craftsmanship vs Software Engineering", which was given at McHenry Software Crafstmanship, Chicago Software Craftsmanship, and South Bend Software Craftsmanship:

Software Craftsmanship VS Software Engineering

View more PowerPoint from AndyMaleh

RailsConf 2012 - My Talk Slides

Here are the slides to my talks at RailsConf 2012.

Software Craftsmanship vs Software Engineering (Lightning Talk)

Rails Engine Patterns

It was nice meeting all of you at the conference. For those looking for a software development opportunity with self-growth potential, hit me up by email (last slide). Cheers!

Software Craftsmanship vs Software Engineering at The South Bend Software Craftsmanship (Apr 2012)

I am presenting on Software Craftsmanship vs Software Engineering at The South Bend Software Craftsmanship on April 17, 2012:
http://www.meetup.com/sobesc/events/59019232/?a=ea1_grp&rv=ea1

Abstract:

The recent emergence of the Software Craftsmanship movement in the last decade has been accompanied with quite a bit of confusion on what the movement is exactly about and whether it adds any value beyond previous software development movements, such as Agile and Software Engineering.

In this talk, Andy Maleh will define Software Craftsmanship, compare and contrast to Software Engineering, and provide actual examples on how both disciplines are playing out at the Groupon software development environment. At the end of the talk, attendees will be given a chance to ask questions and share their own insights on the topic.

Bio: Andy Maleh is a Software Engineer at Groupon who specializes in user needs analysis and building transformative software that meets ongoing demands. He leads by embracing agile practices and software craftsmanship in the process of perfecting Groupon's deal experience. He joined Groupon via Obtiva, where he served as a Senior Consultant for more than five years. Andy is also the Founder and Lead Developer of the Glimmer open source project for Desktop Development with Ruby. Andy holds an M.S. in Software Engineering from DePaul University in Chicago and a B.S. in Computer Science from McGill University (Montreal). Outside of Groupon walls, Andy plays the drums in indie rock venues and travels via Longboard when the Chicago weather permits.

The event is hosted at the Greenhouse at Innovation Park (Notre Dame). 1400 E. Angela Blvd., South Bend, IN, USA.

Rails Engines Patterns at RailsConf 2012

Just a note that I will be speaking soon at RailsConf 2012 (Apr 23-25) about Rails Engines Patterns in Austin, Texas.

Abstract:

This talk covers a successful utilization of Rails Engines to share features that cut across the layers of MVC in different Rails 3 projects. Rails Engines thus provide the best of both worlds: improved productivity by reusing MVC code (including assets like Javascript, CSS, and Images) and better flexibility by allowing different applications to customize behavior as needed without reliance on application-dependent conditionals. Rails Engine patterns will be provided to guide developers on how to leverage Rails Engines' reusability and flexibility without sacrificing maintainability.

Outline:

• Basics of Rails Engines
• Rails Engine Patterns
• Improved Productivity Tips
• Summary of Benefits and Trade-Offs

Attendees should walk away with an overview of Rails Engines and guidelines on how to utilize them effectively.

Hopefully will see some of you blog readers there.

Software Design Trilogy - Part III: Domain Driven Design for RoR Apps

I just gave a talk at Groupon's Geekfest titled:

Software Design Trilogy III - Domain Driven Design for RoR Apps: When Controllers and ActiveRecords Won't Cut It Anymore
You may check out Part II over here.

Here is the abstract of the talk:

Software development is not just about translating business requirements into a static software design. It is about ensuring that in the long run, the system can evolve with the business needs and facilitate communication between business people and developers around the business domain in order to quickly and accurately add or modify features. This requires superb knowledge crunching and domain modeling skills.

Domain Driven Design is an approach that Eric Evans popularized in the mid 2000s to address these concerns. It is a process that centers design of software around the business domain emphasizing the following techniques:

• Domain Modeling and The Ubiquitous Language
• Model Driven Design with Building Blocks
• Refactoring Toward Deeper Insight and Supple Design
• Large Scale Structures, Bounded Contexts, and Domain Distillation

This presentation will cover an example business domain to guide the attendees through several of the topics and ideas in the Domain Driven Design approach. Additionally, there will be a few code examples to illustrate some of the ideas as applied in a Ruby on Rails application, such as how to go beyond just a basic MVC architecture with Model Driven Design.

Domain Driven Design can help in the long run improve large scale code design inspired by the business domain, and enhance developer communication and collaboration with the business stakeholders. Attendees should walk out with a basic understanding of how abstract concepts from Eric Evans' Domain Driven Design book can be applied on an actual project in the enterprise.

And here are the slides:

Software Design Trilogy Part III - Domain Driven Design for Ruby on Rails Applications

View more presentations from AndyMaleh.

This concludes the Software Design Trilogy. Hope you enjoyed the learning.