Agile Processes/Technologies in Non-Agile Hands - Part 1 of 4

One of the things that I encounter often with folks disenchanted with Agile processes/technologies is that they tried them without guidance or proper training (e.g. XP or Ruby on Rails), failed due to reasons mostly related to lack of skill, and then too quickly blamed the failure on the technology and/or Agile process. However, too often what I see when I check their codebase or environment is not Agile, yet Adhoc software development disguised as Agile and sprinkled with a few Agile practices here and there superficially applied.

To be more specific, here are the things I see in their environment:

1. Under-engineered code design with too much coupling between all the different pieces.
2. Difficult-to-maintain test suite.
3. Lack of long term vision for features being developed.
4. Extremely under-optimized code performance.

Now, let's address each of these items in their own blog post, including what the misconceptions Non-Agile folks have about them.

"Under-engineered code design with too much coupling between all the different pieces"

In the Waterfall process, it is recommended for developers to spend time designing the code up-front before diving into implementation. That prompts developers to often think of all the pieces, study coupling and cohesion, and come up with a design flexible enough to handle future requirements. Now, it's important to have software design as a skill for any developer Agile or not in order to ensure the code is maintainable. But, here is where misconceptions come into play with many inexperienced developers. Waterfall developers think that if you design for any possible future requirements, the codebase will be so easy to maintain when the future requirements come that they will be extremely productive. Thinking of it black (extremely engineered) and white (no engineering) like that results in code so over engineered that developing to meet today's business requirements comes to a crawl in terms of productivity since the code design tries to address requirements way far into the future, maybe even 6 months or a year ahead. This in my experience results in code so difficult to maintain that it discourages developers from following good design practices in the long term as they take shortcuts here and there due to business pressure whenever adding new features, resulting in a terrible code base.

People new to Agile that are not skilled with it get a different misconception as a result. They think that Agile is a license to under-architect software for the sake of simplicity, forgetting about Agile incremental evolutionary software design. That works well for the first few weeks of a project, but then very quickly deteriorates to extremely under-engineered software for business requirements, and extremely terrible software performance, either making the developers eventually get disenchanted with Agile, or making developers outside of the Agile community snicker (whenever you snicker know you're risking loss of insight about something) thinking "we are disciplined spending time on architecting for performance and future requirements from the start unlike these folks". Unfortunately, that statement also shows lack of experience with successful sustainable business software delivery as the reality of a system over-architected from the start is like I said before, extreme lack of productivity at the beginning of a project that loses business trust, and then neglect of developers to keep the code design of high quality in the future due to how over-complex it is, resulting in a code base that keeps getting worse in over-complexity till the project needs a rewrite. Though some developers accept that as a fact of software development, in my opinion and experience, that's not true beyond that they choose to write code that makes their life hell and then convince themselves that they are just accepting the "reality" that they themselves created.

Now, to best explain how Agile code design flows through a project, here is a diagram that contrasts code design complexity in Agile over time vs code design complexity flow in Waterfall and Adhoc processes (note that the diagram is used strictly for communication. It is not based on statistically collected data):

Code design complexity goes higher as more structure is added to the code. Think of 0 as an entire code blob lumped together in one file and 12 as code organized into many files along the lines of object oriented domain driven design and design patterns.

There are multiple things to note about the diagram:

1. Complexity in the Adhoc process remains quite low over time as Adhoc developers tend to be averse to advanced design techniques such as object oriented design, inheritance, and design patterns since they consider them over-engineering.
2. Complexity in the Waterfall process tends to go up in spikes since Waterfall developers tend to do big up-front design of software adding a lot of structure way before the need has presented itself.
3. Complexity in the Agile process gradually climbs up following day-to-day business requirement complexity instead of preparing for future business requirements far in advance or simply ignoring complex design techniques in the name of simplification. Notice how the Agile graph overlaps the Adhoc graph in the first 2 months before it diverges later to handle increased complexity in business needs.
4. Complexity in the Agile process tends to have dips that signal how Agile developers occasionally refactor their code to simplify when business requirements have changed and the code has become over-engineered for their current requirements.
5. The area in the diagram between the Waterfall graph and Agile graph represents lost productivity by Waterfall developers due to having to work around over-engineered code.
6. The area in the diagram between the Adhoc graph and Agile graph represents lost productivity by Adhoc developers due to having to work too hard with weak or non-existant abstractions (under-engineering) that makes them have to deal with difficult to detangle code (lack of separation of concerns).

Now, all of this might be common sense to a lot of developers. But, it takes a lot of practice to get disciplined at refactoring your code design regularly and boldly (with the protection of automated tests) to ensure it is neither over-engineered nor under-engineered per today's business requirements.

Still, there are quite a few other misconceptions that I would like to address, such as:

• This all sounds good, but in "reality" developers do not have time to refactor their code regularly while still meeting goals for today's business requirements: This is like saying "All the health professional talk about food and exercise sounds good, but who has time to eat well and exercise while still being able to earn their buck for their families". Indeed, this can be a dilemma for many people, and I do not deny it. I am not the most disciplined at exercising for example. But, this is not how I think of it. The way I like to think of it is more along the lines of: "If I were to actually eat well and exercise, would I perform better in other areas of life?". Since the answer is yes, I trained myself to appreciate the taste of healthy food, thus eventually effortlessly consuming such food out of habit. I also figured out alternative ways to exercising like snowboarding, frequent longboarding, and playing the drums, keeping myself in shape, again effortlessly. Going back to our original point, "If I were to become very disciplined at refactoring code regularly, will I be better able to meet business demands in the long term?". The answer from my experience is a most certain yes. In fact, you end up meeting the goals of today's business requirements much more often in the long term if you have a code base that is neither over-engineered nor under-engineered for today's needs. So, the key thing in the short term is to keep practicing the skills of Agile software development until you effortlessly develop the habit of just-enough-design and disciplined refactoring.
• What you describe as helpful in increased software design complexity is not really helpful for me. I never quite got the point of object oriented design, and I like to break my code into many small methods to handle complexity: While code design is certainly subjective to an extent. When working with a large code base that will be maintained by many future developers. In my experience, organizing code in an object oriented fashion around the business domain concepts helps free my mind from thinking about blobs of code to focus on abstract concepts that relate to business directly, thus better manage complexity. It takes quite the skill to think of code as abstract concepts instead of low level data shuffling with for loops and if loops, bringing us back to the idea that this misconception might be more due to lack of skill in object oriented design as opposed to a problem with the methodology itself. The same way structured procedural design is a step-up from being able to code if statements and loops in one procedure effectively, object oriented design is a step up for business application development in my opinion from structured procedural design, and it assumes you are quite skilled in structured procedural design as a pre-requisite. Of course, there are domains that benefit better from other paradigms, like the functional programming paradigm for mathematical computing and logic paradigm for boolean algebra, which is why I am recommending object oriented programming specifically for domain business model related code that has gone far in complexity beyond the manageability of structured procedural programming. If you find yourself uncomfortable with object oriented design, I strongly recommend getting more training in it, especially from people that are skilled at applying it in practical enterprise environments as opposed to class room settings.

My next blog post will focus on "Difficult-to-maintain test suite.". Stay tuned for part 2.