Sunday, March 07, 2010

Revisiting Software Craftsmanship vs Engineering

With the emerging software craftsmanship movement, I have been noticing a lot of confusion out there as to what it is, whether it has any value at all or is just another fad, and whether it is the new silver bullet that trumps the older ways.

To help sort some of the confusion out, I wrote a blog post last year titled: Software Craftsmanship vs Software Engineering.

Did my opinion change since that blog post?

So much has changed in the craftsmanship world since then, especially given that Obtiva, the Chicago agile consulting company I work with, hosted the first Software Craftsmanship conference here in Chicago, which ran at the same time as the Agile 2009 conference. And as a result, I certainly learned a few new things about the differences and similarities between software craftsmanship and engineering.

Let's start with the summary of last year's blog post:
... Software Engineering is not about over-engineering or heavy-weight processes like Waterfall; it is about delivering economical software that is reliable and efficient. Software Craftsmanship is therefore not in conflict with Software Engineering, yet in fact in harmony with it as it helps accomplish its goal through practical learning of software development with the guidance of experienced developers.
Since that conclusion, a few more differences have emerged:
  • Engineers often seem to believe that they can completely streamline the process of building software, with complete control and predictability. And, this belief seems to be process independent. In other words, even engineers who are strong advocates of agile often believe that if you do a, b, and c, you will get the results you want.
  • Engineers often think that one has to learn all, most, or many best practices before successfully building software.
  • Some engineers seem to think that best practices apply everywhere or in every situation. More experienced engineers are aware that different contexts require different best practices though.
On the other hand, here are some observations I made from observing developers who have not had much software engineering background and thus have a bit more laissez-faire attitude that leans towards learning your skills through craftsmanship as opposed to a systematic study of best practices:
  • Craftsmen see software development more as an art that emerges and less as a science that can be controlled, so they let time and experience shape up their skills in successfully completing software projects.
  • Craftsmen often discover their own best practices from their experiences, which often better fit their situations.
  • Craftsmen do not religiously follow even their own best practices, often transcending them when they do not make sense in a new situation anymore. They thus rely more on their intuition and gut feelings to succeed.
It was not easy for me to spot these differences. They only started to become apparent after observing the way our top craftsmen at Obtiva have worked over the last year. After all, they often relied more on common sense and experienced intuition than industry best practices, and yet still succeeded at delivery as demonstrated by Mad Mimi ( for example.

Even Tom DeMarco, one of the early key figures in the software engineering world, famous for the quote "You cannot control what you cannot measure", recently renounced the controlled development approach in an article posted by the IEEE, titled "Software Engineering: An Idea Whose Time Has Come and Gone?"

Although I noticed these differences, let me clarify that nothing in software engineering's original definition or goals necessarily states that you must follow certain best practices or processes to succeed. It just happened that people who adopted the software engineering approach tended to gravitate more towards a controlled measured fashion to software development.

For example, one of the common practices in eXtreme Programming is to develop features within iterations and then measure the velocity at the end of each iteration in order to determine your velocity and have a better estimate for your stakeholders. Software engineers following that process would often take that practice as gospel and believe that you will not succeed unless you build your software iteratively and measure velocity consistently. I know I have made that assumption in the past, and did not change my mind till I saw projects that succeeded without any iterative scheduling or measuring.

As a software engineer who likes to question everything, I definitely wondered why. If I were to solve this following the software engineering discipline, I may ask what is the cost of iterative planning and what are the benefits offered by it? Then, I would check our current environment and see if it fits the cost/benefit of iterative planning. Though this is an important software engineering practice, many engineers often forget about it and just follow a process blindly out of comfort and past experience with it.

To bring it back to my example, if the project is in the maintenance phase and all what the stakeholders want is bug fixes and tiny feature additions here and there, then why would they need to know the team velocity or do iterative development at all? The Kanban approach, which loses them the benefit of more accurate estimation, may fit them better since most software changes at that point would take no longer than two weeks anyways if not just a couple of days.

However, as you notice, I was able to figure that out by applying software engineering disciplines only without having craftsmanship come into play. So, how does software craftsmanship fit into the story?

Well, as stated before, software craftsmen often learn their best practices from their environment, their personal experiences, and their mentor's experiences. So, if an apprentice worked in some environment where most projects were in maintenance mode and far-term planning was rarely necessary, they would naturally succeed without any iterative development or velocity measuring, which may potentially baffle even some of the most experienced software engineers. However, the moment they hit a green field project, they will realize that their older methods are not working anymore because the stakeholders now need to know how long the next 20+ features will take for budget reasons. That is when they are forced to learn a new estimation approach, such as iterative velocity-based estimation.

So whereas software engineering teaches you to learn as many best practices as available out there and gain the skill of analyzing any situation objectively to prescriptively apply the best practice that fits, software craftsmanship actually encourages you to learn best practices gradually out of your own experiences and mentor experiences, and thus naturally apply the best practices that fit without much analysis.

So, what are the trade-offs between these two approaches?

  • With software engineering, you have to work really hard at learning processes, best practices, and analysis techniques, and then spend quite some time gaining applied experience, before you can be a good software engineer. However, once you have gone through that hump, you have a very wide perspective on how software can be developed, which will make your opinion quite valuable to your team.
  • On the other hand, with software craftsmanship, you can start developing production quality software much quicker by apprenticing with a few experienced craftsmen and learning the ways that worked for them and their projects. That will keep you motivated to keep learning and thus widening your perspective gradually to the different software development methods out there. The trade-off though is you may end up having to fail really hard at applying a practice that does not work anymore in a new situation before you learn another way. That happens even though the better way may have been discovered 15 years ago by other people who have faced the same situation long ago.

In any case, one of the things that experience has taught me over the years is to try to stay as unbiased and detached as possible from any approach instead of defining myself through one. So although I have a master degree in software engineering, that does not stop me from learning the ways of software craftsmanship or becoming a software craftsman myself. And, I encourage you to do the same.


Dave Hoover said...

A very well thought out post, Andy. I've always admired your tenacious appetite for learning.

Alfredo Perez said...

Really nice post. Very complete.

Madhu said...

Craftsmanship will bring the best out of the engineer!!!

Great Post!

Paul W Pagel said...

Great Post.

Is there a chicken and egg problem with learning best practices from your mentor and experiences though due to the lack of diversity? How would you ensure you have enough exposure to know your experiences are following the right paths?