Any Good in Learning Big Up Front Design?

Since the beginning of the Agile movement, people have been renouncing big up front design in favor of incremental emerging design of software. It occurred mostly as a reaction to overly complex designs and application architectures that slowed down productivity to a crawl, especially on new projects. Anyone remembers the days of EJB 2.1? You had to configure a handful of XML file descriptors and write quite a bit of code following the EJB 2.1 conventions before you got anywhere on a new project. Now, contrast that with the Ruby on Rails architecture and how it enabled developers to get a CRUD web application up and running within minutes. Most people miss out on the other side of the story though, that is the part about why EJB's architecture was so complex to begin with. After all, development with a technology like Java's JSP alone was a lot simpler.

EJB came out as a reaction to insecure badly written Java web applications that did not take advantage of transactions and threading correctly, and mixed data persistence concerns with web flow concerns. Enterprise JavaBeans's main innovation was aspect orientation of concerns like transactions, security, threading, and caching, enabling Developers to get their benefits without coding them directly. All they had to do is configure the XML descriptors to enable them, and voila! Additionally, Entity Beans were of the earliest forms of ORM (Object Relational Mapping), Session Beans were one of the earliest forms of controllers, and Message Driven Beans were one of the earliest forms of web background workers (think Resque in the Ruby on Rails world). Also, Enterprise JavaBeans allowed nice separation between web flow and data persistance at the time (albeit with non-existent OO inheritance support), providing one of the most primitive versions of Web-MVC.

Now, if you look at many of today's web frameworks, like Rails for example, you notice that they got many of the same features that EJBs innovated, such as easy transaction support, ORM, security (authentication via Devise and authorization via CanCan for Rails), threading (automatic spawning of threads for requests in web servers like Phusion Passenger), and caching.

Sure, if you needed something much simpler, you can start a Rails app without any of the complexities of caching or other advanced features getting in the way. You may even use Sinatra and avoid reverting to Rails until your app grows large enough to warrant the shift in complexity of design.

However, if people did not spend time solving the problems plaguing web applications in the early 2000s, doing some big up front design, we might have not had any of the solutions that can help an Agile business scale up with Rails today once they have more than a handful visitors an hour.

How does this idea transfer to software design?

Developers new to object oriented programming often encounter topics such as inheritance vs composition and design patterns. If they were to bypass such topics with excuses like "inheritance produces overly complex code over procedural code reuse" or "design patterns make for overly complex designs", their blaming of the tools is sure a sign that they are not skilled with them and that they use them like a golden hammer, potentially in the wrong situations. Now, if they were to avoid learning them however, and they eventually work on a business domain that grows enough in complexity to a level where switch/giant-if statements are plaguing the code everywhere (lack of use of object oriented polymorphism/inheritance or design patterns), given that they skipped learning the techniques of big up front design, they will unfortunately fail in incremental designs as they will always stick to the primitive design techniques they started programming with (like structured decomposition of logic into methods) and thus not be able to scale up their code base to handle more complexity while remaining clear and maintainable to their fellow programmers.

That is why I strongly suggest to developers spending time learning many of the big up front design techniques out there, such as Responsibility Driven Design, Domain Driven Design, Design Patterns, Architectural Patterns, UML Modeling, and even Model Based Development (foundation behind use of DSLs) , simply for the sake of expanding their toolset for when their small simple solutions no longer provide high productivity and maintainability for the increased complexity of their project. In other words, practicing big up front design in a safe learning setting enables developers to actually become (Oh My!) more Agile when the time comes for employing the advanced design skills.

I am sure I missed a handful of useful design tools out there, so I would like others to pitch in. What design tools would you recommend to others to learn and employ in their toolset?