Cheaper Than The Cost of Failure

Applying certain quality practices for software development may not be cheap, but is cheaper than the cost of failure, whether it is TDD (Test-Driven Development), pair-programming, or usability testing.

Let's talk about TDD for example. Although it significantly decreases the cost of building reliable software for developers experienced with the technique, TDD requires a 3 month learning curve for a team new to it before they can start becoming more productive with it than without it from my experience. And, that's with the help of an experienced TDD coach. Now that may seem like a prohibitive cost when business development is in full swing demanding that developers deliver yesterday. That is why often the teams that hire Agile consulting folks are the ones who are already in dire trouble with their managers for being unable to deliver quality software on schedule. They are often stuck in bug fixing mode for both older and newer features, and thus are willing to pay the cost of learning better software development practices. They may resist suggested practices like TDD at first, but then when they see that software releases requiring weeks of bug fixing pretty much fail to wow customers or gain the trust of their stakeholders, that's when they realize that the cost of learning TDD may not be cheap, but is a hell of a lot cheaper than the cost of failure!

Usability testing is another example. I worked on a project several years ago where the stakeholders were exploring new user interface ideas mixing social networking with education. That seemed like the perfect situation to benefit from usability testing with paper prototypes. We tried the process for a feature or two and noticed that it was taking us about 3-4 hours per feature to come up with the paper prototypes and test them given our lack of experience with the technique. The cost of it seemed prohibitive for stakeholders, so management decided to drop the practice. Six months later, when the project was slated for release, stakeholders realized many inefficiencies in the user interface with beta-testing and wanted developers to rework certain areas completely. Unfortunately, that took a lot longer than revising user interfaces in paper prototype format (think 3-5 days vs 2-4 hours per feature) and investors had almost ran out of money by then, so they had to release the project the way it was. The released website did not end up as revolutionary as they originally intended it to be, and business had to accept the inefficient first-cut designs they had for most of the user interfaces.

Morale of the story? The next time, people try to pressure you into giving up certain practices just because of the cost associated with them, emphasize the importance of the benefits and tell them "It may not be cheap, but it's cheaper than the cost of failure!"

Influential Books On My Career in Software Development

I was just thinking about how much influence certain books had on my career, whether concerning object-oriented design, enterprise architecture, or even business soft skills, so I decided to list them in this post.

Here are the technology books that had the most influence:

• The Pragmatic Programmer: I still remember the fascination I had with this book after reading the story about Stone Soup in the first few chapters. It was one of those soft skill patterns that come in extremely handy when you think you have a great idea to improve effectiveness at work, but are faced with a lot of resistance to it by coworkers (especially novel ideas like Pair-Programming). In summary, the lesson from Stone Soup is to walk the walk first, and then let people talk the talk for you, instead of starting with the talk before any walk. You start applying the idea at a very small scale, and if it demonstrates benefits, people testify for you, encouraging the application of the idea on a larger scale. And if not, no harm was done given the small scale you started with. You got to try the idea at least, and can rule it out moving on to another idea instead of dwelling on people's resistance to talking about it with no action taken. I found this book a great read when I stumbled upon it several years ago as it contains many lessons in soft skills in addition to technical skills as developers need both to succeed at work after all.
• Head First Design Patterns: This was one of the most entertaining technology book reads I have ever had, and I still find the comedy in it extremely enjoyable and educational. It is my favorite on how to learn practical application of Design Patterns, and by practical I mean, knowing when it is appropriate to apply them, not just how to apply them. 
• Applying UML and Patterns: This book has been perhaps one of the most influential books on my object-oriented design skills as well as integrating the process with Agile software development. And, don't let the name of the book fool you. It is not focused mainly on applying UML. It just emphasizes Agile UML diagramming (mostly on paper or whiteboards) as a healthy communication mechanism between team members to better brainstorm and get ideas across. GRASP Patterns (e.g. Coupling and Cohesion) are a huge takeaway in this book as they serve as the underpinnings of successful object-oriented design, and knowing when design patterns are beneficial. 
• Domain Driven Design: One cannot do object-oriented design well in isolation from business. Knowing how to speak the language of the business domain is crucial in achieving an easy-to-maintain-and-extend software design. 
• Refactoring to Patterns: Design Patterns are often best applied gradually in response to new business needs as opposed to prescriptively without the business needs being present yet. For example, if an application only seems to need two ways of sorting data on a report at the moment, a simple "if" statement might get the job done well. If in the future, new business demands required more sorting options that need to be reused in multiple parts of the application, then sticking with the same code design simply expands the use of the "if" statement with more branches. That makes readability of the code less abstract and understandable at a glance, and requires duplication of the "if" statement in all parts of the app that require the same sorting options (unless procedural method reuse was done). That is an example of a need to refactor the code to the Strategy design pattern in order to replace the if statement with reliance on polymorphism (GRASP Pattern) via sort strategy objects. "Refactoring to Patterns" is a great read on this extremely useful Agile skill, and so is its predecessor Refactoring.
• XP Explained: Reference book on the radical ideas of eXtreme Programming. A must read for any software developer who wants to go beyond the overly-controlling Waterfall process or out-of-control adhoc process in software development, even if they cannot apply all its ideas in their environment. It is eXtremely mind expanding as it pushes your thinking outside the boring box that the masses tend to constrain their thinking to.
• Apprenticeship Patterns: Awesome book on how to become a quick learner (a must in the technology field), stay humble despite experience, and remain open to new ideas the way beginners are. Some of my favorite patterns are "Wear the White Belt", "Expose Your Ignorance", and "Breakable Toys".

Notice how I did not mention any books on Test-Driven Development or Pair-Programming. That is because from my experience, the best way to learn them is not through reading books, yet through actual daily experience via pair-programming with other developers (apprenticeship) that are quite experienced with both skills (these two skills go hand in hand). I have seen developers inexperienced with test-driven development miserably produce difficult to maintain test suites with complex code implementations that are difficult to refactor. I have also seen developers nod off or waste time tweeting while pretending to pair-program. To avoid such pitfalls, it is extremely important to pair-program with experienced pair-programmers, and to learn all different styles of test-driven development (e.g. state-based vs interaction-based, outside-in vs inside-out, integration vs unit, etc...) to understand their trade-offs and when to apply each. After all, badly done Test-Driven Development and Pair-Programming is like badly done anything. It will result in worse results, and unless the developer had enough foresight, they might then blame the technique instead of their lack of skill in it, and thus miss out on the benefits of learning completely. Just think, a non-skilled skiier might not enjoy skiing very much. But, is it the activity of skiing or their lack of skill in it that is responsible (assuming they are quite interested in it)?

Here are a few non-technology books that also had a huge influence on my success at work:

• The 7 Habits of Highly Effective People: Helped me direct my career and sort my priorities as taught in the habits "Begin With The End In Mind" and "First Things First". Also, it had a huge influence on the way I try to come up with solutions that bring everybody in the company together, whether in development, QA, or business  (the "Think Win-Win" habit) and be open minded to collaboration that achieves much more creative solutions to problems than flying solo (the "Synergy" habit)
• How to Win Friends and Influence People: Great aid in social skills for a techie like me, helping with public speaking skills and learning respect for non-autonomous and barely-logical entities (a.k.a non-computers, a.k.a. human beings!) 
• The Magic of Thinking Big: This was the book that shattered all artificial and societal constraints that I had acquired over the years about how much I can achieve in life. After reading it, I stopped believing that I am constrained to my I.Q. (or maybe realized you can increase your I.Q. indefinitely) and stopped seeing any boundaries to anything I can do at any organization or in life in general for that matter. Let's just say that not only did I eventually learn to do public speaking in front of masses of people (EclipseCon, EclipseWorld, RubyConf, etc...) after a quarter century of being a shielded introverted shy person, but I also learned rudimentary drumming skills in the last 3 years and now perform in two rock bands (Gag Order and Cletus Darby) around the city of Chicago. 

What are the books that had the most positive influence on your career?

Any Good in Learning Big Up Front Design?

Since the beginning of the Agile movement, people have been renouncing big up front design in favor of incremental emerging design of software. It occurred mostly as a reaction to overly complex designs and application architectures that slowed down productivity to a crawl, especially on new projects. Anyone remembers the days of EJB 2.1? You had to configure a handful of XML file descriptors and write quite a bit of code following the EJB 2.1 conventions before you got anywhere on a new project. Now, contrast that with the Ruby on Rails architecture and how it enabled developers to get a CRUD web application up and running within minutes. Most people miss out on the other side of the story though, that is the part about why EJB's architecture was so complex to begin with. After all, development with a technology like Java's JSP alone was a lot simpler.

EJB came out as a reaction to insecure badly written Java web applications that did not take advantage of transactions and threading correctly, and mixed data persistence concerns with web flow concerns. Enterprise JavaBeans's main innovation was aspect orientation of concerns like transactions, security, threading, and caching, enabling Developers to get their benefits without coding them directly. All they had to do is configure the XML descriptors to enable them, and voila! Additionally, Entity Beans were of the earliest forms of ORM (Object Relational Mapping), Session Beans were one of the earliest forms of controllers, and Message Driven Beans were one of the earliest forms of web background workers (think Resque in the Ruby on Rails world). Also, Enterprise JavaBeans allowed nice separation between web flow and data persistance at the time (albeit with non-existent OO inheritance support), providing one of the most primitive versions of Web-MVC.

Now, if you look at many of today's web frameworks, like Rails for example, you notice that they got many of the same features that EJBs innovated, such as easy transaction support, ORM, security (authentication via Devise and authorization via CanCan for Rails), threading (automatic spawning of threads for requests in web servers like Phusion Passenger), and caching.

Sure, if you needed something much simpler, you can start a Rails app without any of the complexities of caching or other advanced features getting in the way. You may even use Sinatra and avoid reverting to Rails until your app grows large enough to warrant the shift in complexity of design.

However, if people did not spend time solving the problems plaguing web applications in the early 2000s, doing some big up front design, we might have not had any of the solutions that can help an Agile business scale up with Rails today once they have more than a handful visitors an hour.

How does this idea transfer to software design?

Developers new to object oriented programming often encounter topics such as inheritance vs composition and design patterns. If they were to bypass such topics with excuses like "inheritance produces overly complex code over procedural code reuse" or "design patterns make for overly complex designs", their blaming of the tools is sure a sign that they are not skilled with them and that they use them like a golden hammer, potentially in the wrong situations. Now, if they were to avoid learning them however, and they eventually work on a business domain that grows enough in complexity to a level where switch/giant-if statements are plaguing the code everywhere (lack of use of object oriented polymorphism/inheritance or design patterns), given that they skipped learning the techniques of big up front design, they will unfortunately fail in incremental designs as they will always stick to the primitive design techniques they started programming with (like structured decomposition of logic into methods) and thus not be able to scale up their code base to handle more complexity while remaining clear and maintainable to their fellow programmers.

That is why I strongly suggest to developers spending time learning many of the big up front design techniques out there, such as Responsibility Driven Design, Domain Driven Design, Design Patterns, Architectural Patterns, UML Modeling, and even Model Based Development (foundation behind use of DSLs) , simply for the sake of expanding their toolset for when their small simple solutions no longer provide high productivity and maintainability for the increased complexity of their project. In other words, practicing big up front design in a safe learning setting enables developers to actually become (Oh My!) more Agile when the time comes for employing the advanced design skills.

I am sure I missed a handful of useful design tools out there, so I would like others to pitch in. What design tools would you recommend to others to learn and employ in their toolset?

Rails Flash for Presentation Logic: Good Idea or Cause for Trouble?

Rails comes with a feature called Flash that simplifies displaying messages to users after redirecting them to another page on the site. Such messages can include confirmations, greetings, congratulations, etc..

For example, upon a user submitting a help request on the site, they see a flash message on the next page confirming that the request has been submitted and is being reviewed.

The HelpRequestsController create action code would contain logic like the following:

if help_request.save
  flash[:notice] = 'Your help request has been submitted and is being reviewed.'
  redirect_to root_path
else
  flash[:error] = 'The help request is missing required information.'
  render :action => 'new'
end

This would display a view element as follows, typically at the top of the page:

The key benefit to using the flash hash is that its data persists whether the controller action renders a page in the same request, or redirects to another page initializing a new request. It relies on the session store behind the scenes to persist data and clear the flash after the next request, saving the developer from performing that work themselves.

Now, this blog post is not about use of the flash hash for displaying messages. It is about whether it is a good idea to rely on the flash ability to keep data for one more request to store data that will affect presentation logic.

I recently saw an example of using the flash to store variables that get rendered in hidden fields and later used by JavaScript logic on the page for determining whether to display certain pieces of data or not.

The logic unfortunately broke the moment someone Ajaxified the JavaScript logic as it ended up hitting another controller action that was not redirected to from the original action that requested the page, thus losing the flash data in the process.

To illustrate:

Originally:

Action 1 sets flash and redirects to Action 2
Action 2 uses flash in rendering elements

After Ajaxification:

Action 1 sets flash and redirects to Action 2
Action 2 renders a page that performs an Ajax request in its JavaScript
Action 3 attempts to use flash in rendering elements but does not find data

One way to fix the problem that a developer attempted was to use the flash.keep method in the last redirected action to keep the flash data for one more request. Unfortunately, that approach is fickle because if multiple Ajax requests hit the server, the flash data will get lost eventually unless all their actions perform a flash.keep. Also, if the code ends up performing flash.keep everywhere, that defeats the purpose of using flash over the session.

My understanding is the flash hash was created to flash a message to the user once after a certain request. So, I personally limit its use only to that use case to keep any code around it easily understandable and maintainable by other developers. If I needed to persist presentation logic data across requests, I usually rely on the session instead.

What is your opinion on the matter?

Decoupling Views from Controllers in Rails (Smalltalk MVC Style)

Updated on 2022-09-23: mostly clarifications for the latest versions of Rails

Last week, I gave a talk at the Groupon weekly GeekFest event titled "Smalltalk MVC Applied in Rails". Though the talk briefly touched upon Smalltalk's original MVC pattern and how it is applied in desktop development, the main focus of the talk was on decoupling views from controllers in Rails the way they are decoupled in Smalltalk MVC, so I would like to elaborate more on that in this blog post.

The diagram above shows the relationship between Model, View, and Controller in Smalltalk MVC with desktop development.

• The Controller observes the View for changes caused by user interactions (e.g. clicking a button, making a selectiong, etc...)
• The Controller causes updates in the Model when it receives a notification from the View
• The View observes the Model for changes caused by invokations from the Controller (e.g. add a new contact) or by Model change events (e.g. a customer birthday has been reached in system time)
• The View refreshes itself automatically from the appropriate models it is providing a view of.

Notice that nowhere in that flow does the Controller directly interact with the View to update it with Model data. In fact, well written desktop applications avoid that sort of coupling to ensure clean separation between control flow and presentation logic.

One thing to note also about Smalltalk MVC on the desktop is that Views are objects in desktop applications with the intelligence of any other objects. Like Models, they are responsible for maintaining their own state as per the object oriented paradigm. They do not require another class like a Controller to update them. However, when presentation logic gets complex enough, it is a good practice to then split that into a Presenter layer between Views and Models that follows the Adapter design pattern, adapting View state (e.g. index of contact selected in a list of contacts) into Models (the actual contact object representing the index)

Now, if we were to transfer all of these ideas transparently to the web, View state is simply the parameters (request or session) that are populated by user actions. Controllers get access to them temporarily on user actions to cause updates in Models, but then once they render a View, the View itself can be responsible for translating its state (parameters) into Model objects via a Presenter layer. The Presenter layer in Rails is nothing but the good old (badly named) Rails Helpers. They automatically get access to the View context (request and session parameters), allowing them to act as Adapters that neatly hide the details of converting request and session parameters into Model objects the View can rely on to render its contents. This frees Controllers to focus only on Model updates and routing control logic, avoiding the typical clutter with Model loading logic that we often see in Rails applications. This then decouples the Views from Controllers the way they are in original Smalltalk MVC.

Here is an example of typical Rails MVC code:


Controller:

def show
  @contact = Contact.find(params[:id])
  @region = Region.find(session[:region_id])
  @friend_contacts = @contact.friends_by(@region)
  @news = News.latest
end

View:

_contact.html.erb:

Name: <%= @contact.first_name %>
Last Name: <%= @contact.last_name %>
Phone: <%= @contact.phone_number %>
...

_region_header.html.erb:

<%= @region.name %>
<%= @region.state %>
<%= @region.city %>
...

_friends.html.erb:

Friends
<% @friend_contacts.each do |friend_contact| %>
  <%= link_to friend_contact.name, contact_path(friend_contact) %>
<% end %>
...

_news_feed.html.erb:

Latest Happenings:
<% @news.each do |news_feed_item| %>
  <%= news_feed_item.story %>
<% end %>
...

Here is the same example benefiting from the Smalltalk MVC pattern:

Controller:

def show
end

Helper (Presenter):

def contact
  Contact.find(params[:id])
end

def region
  Region.find(session[:region_id])
end

def friend_contacts
  contact.friends_by(region)
end

View:

_contact.html.erb:

Name: <%= contact.first_name %>
Last Name: <%= contact.last_name %>
Phone: <%= contact.phone_number %>
...

_region_header.html.erb:

<%= region.name %>
<%= region.state %>
<%= region.city %>
...

_friends.html.erb:

Friends
<% friend_contacts.each do |friend_contact| %>
  <%= link_to friend_contact.name, contact_path(friend_contact) %>
<% end %>
...

_news_feed.html.erb:

Latest Happenings:
<% News.latest.each do |news_feed_item| %>
  <%= news_feed_item.story %>
<% end %>
...


Notice how the last partial did not even need a Presenter and went to the Model directly since it did not rely on any specific View state (parameters).

Of course, you might want to memoize helpers if you use repeatedly in the view:

Helper (Presenter) with memoization:

def contact
   @contact ||= Contact.find(params[:id])

end

def region
   @region ||= Region.find(session[:region_id])
end

def friend_contacts
   @friend_contacts ||= contact.friends_by(region)

end

Here is a summary of the benefits of decoupling Views from Controllers by allowing them to refresh their data directly from Models as per Smalltalk MVC or use Helpers as Presenters/Adapters for View state (request and session parameters):

1. Unclutter Controllers from data loading logic for multiple objects that the View needs, allowing each part of the View to load its data directly.
2. Make View partials easily reusable as they rely on Presenters/Adapters (Helpers) to load their data by pull instead of having to include code in every reusing Controller to push the data into the Views.
3. Easily test-drive and maintain the logic of Presenting/Adapting View data in small cleanly separated methods instead of having that logic all mixed in Controllers.
4. When Models needed for the View have dependencies in their load order, there is no need to explicitly order their loading in the Controller. Helper methods can be composed of other helper methods, resolving the dependencies automatically.
5. Avoid the dissonance in View code caused by a mix of "@object" references and Helper "object" references. All objects in the View get populated from Helpers with "object" references or directly from model classes making the code more readable.
6. Trivial extraction of partials from Views given that they do not contain any "@object" variables and all references are "object" references. Developers thus do not need to put any effort into error-prone switching of "@object" variables into "object" locals. The helper "object" references can already serve as locals.
7. Controllers already have access to the context of Presenters (Helpers) thus are able to reuse the View data loading logic without the need to duplicate.
8. If multiple Controller actions and Views rely on the same data object being present as in the new, create, edit, update, and show actions (e.g. contact object). A single Presenter method (e.g. "contact" helper) can take care of loading that object regardless of whether new or existing in the database already (e.g. Contact.find_or_initialize_by_id(params[:id]) [more recently Contact.find_or_initialize_by(id: params[:id])] )

These are some of the benefits experienced in my last three Rails projects, giving the team great flexibility in maintaining Views, Models, Controllers, and Presenters without the mix of concerns typically experienced in Controllers, allowing for much easier test-driven development and flexibility in composing/modifying features for customers.

In fact, by following Smalltalk MVC correctly in Rails, you would not need any extraneous "View Component" libraries because you would already be building view components natively as demonstrated by the presenter examples above, which can either be well-factored helpers or partials. Such extraneous libraries only add layers of unnecessary complexity due to a misunderstanding of how to follow Smalltalk MVC correctly in the first place.

p.s. The "region" Helper above can optionally be enhanced to manage the region session state in isolation of any Controller, thus maximizing reuse for the Views relying on that View state (parallel to how desktop Views manage their own on-going state as smart objects). Here is one of several ways to do this:

def region
  session[:region_id] = params[:region_id] if params[:region_id]
  Region.find(session[:region_id])
end

p.s.2 Though "before_filter"s in Rails (more recently "before_action"s) can be used to easily load data in controllers, they still put the onus on the controller to do the data loading by push, requiring developers to add such logic to every conroller that will reuse a particular View partial, and adding complexity to reasoning about the code.

--

If you have any questions, remarks, or corrections, feel free to share in comments.