Software Development VS Software Manufacturing
Embracing The Nature of Software Development to Deliver Valuable Software Faster
Software development is a term you don’t think much about. But its meaning is not well understood. Do companies today really practice software development or software manufacturing? Are leaders and managers of the software industry embracing the nature of software development? Let’s start with a bit of history.
The age of mass production has had a great influence on how companies operate until today. Frederick W. Taylor invented a management system called scientific management. Also known as Taylorism. In his view, managers and consultants should determine the only right way to work for factory workers.
Workers, who are considered replaceable resources, should only follow this way, like cogs in a machine. No talk, thought, or collaboration is required; only following the only right way.
In his view, the main objectives of this system are to improve economic efficiency and labor productivity. Notice the complete ignoring of the human element in this system.
Henry Laurence Gantt, the creator of the Gantt chart, worked closely with Taylor and was a promoter of Taylor's teaching and the scientific management method. They worked together at Midvale Steel company, Simonds Rolling company, and Bethlehem Steel Corporation.
The Gantt chart is a timeline used to track the progress of workers toward a task. Typically, every detail was recorded on this chart, with start and end dates. For a steel company, that might look like this:
The teaching of Frederick W. Taylor and Henry Laurence Gantt had a great influence on the companies they worked with and on many managerial books that came after.
Technology was an emerging field in the 80s and 90s. Companies started to hire programmers. The pervasive management style at that time was Taylorism. So, they apply it to programmers too. This is the beginning of Waterfall development.
In waterfall, managers, consultants, analysts, and/or architects write all kinds of documentation and specifications. Programmers then write code following these documents. Project managers oversee the projects, ensure the documentation is followed, and allocate the necessary resources (people) considering the allocated time and budget. Like managers of the mass production age, project managers also use the Gantt chart, but instead of cutting and moving steel, they use it for the activities done by the various and typically siloed teams in IT:
The result of these practices is a crisis in the software industry. An infamous CHAOS report by the Standish Group reported that only 16% of software projects were completed on time and on budget. This is a staggeringly low percentage.
The Reason These Practices Don't Work in Software
In the physical world, when engineers embark on making a new product, there are typically two phases, the design phase and the production phase (or the development phase and manufacturing phase).
In the design phase, engineers might start with research and sketching some ideas. They might create low-fidelity prototypes to test their hypothesis. Then they might create high-fidelity prototypes to get the measurements and specifications right. They get feedback and redo the research and prototypes as necessary. After reaching a certain confidence level, they start the mass production process.
Let's say the engineers want to make a coffee mug. There are plenty of mugs in the market, and they wish to make something different. They designed an interesting mug handle and made it heat-resistant. After they built the prototype and tried the mug with potential customers, they found that the mug handle was not comfortable to hold, so they altered the design to a simpler one.
Now the handle is more comfortable to hold, but doing another round of research with customers, they found the mug is too heavy because of the heat-resistant material. After talking to more customers, they found that the customers aren't concerned with the mug being heat-resistant; since the mug has a handle, not many customers are interested in grabbing the mug without using the handle. So, the engineers decided to remove the whole heat-resistant material from the mug (a whole feature) to make the mug more comfortable to hold and less expensive as an additional advantage.
Now they started to receive more positive feedback from their prototypes and decided to mass-produce this mug.
In the production phase, a factory that is now responsible for creating this product (or widget in manufacturing terms) will need to produce this widget with exact specifications hundreds, thousands, or probably millions of times. It's a replicating process. The Taylorism management style might work in the production phase since the factory is creating exactly the same coffee mug repeatedly. In this situation, the predictability is high.
But imagine doing this management style to the engineers in the design phase. Engineers still need to learn what exactly a good coffee mug is. Imaging doing this Gantt chart for the design phase:
What if engineers, after creating the low-fidelity prototype, found that they had to change something in their plan? In the Taylorism approach, they are not allowed to do that, they must proceed to build the high-fidelity prototype, and by the 6th of July, they must deliver the final specifications. After all, we have a deadline; we need to give the specifications to the factory to start the production process, right?
There is no value in producing the wrong thing, No one wants to use an expensive and heavy coffee mug that is unnecessarily heat-resistance since it has a handle. That's why you typically don't see this approach in the design phase. Instead, engineers work on iterations. Yes, they plan and build prototypes, but they test things out and talk to potential customers, they build prototypes again from low-fidelity to high-fidelity and vice versa and talk to customers again. They only stop after validating that they discovered the right product that will solve a real customer problem. Only then do they proceed to the production phase and start mass-producing this valuable product.
In the software industry, we only have the design phase. The production phase, lucky for us, is almost free.
To help you understand why the production phase is free in software, imagine this. If we need to develop a log-in page for our app or website, we don't plan, design, code, and deploy this page to one customer, then plan, design, code, and deploy it to the second customer, then again for the third customer, and so on. Absolutely no.
First, we need to ensure we build a valuable login page that our customer needs. Do we let the customer login with a mobile number or an email? It depends. Should we require the customer to enter a hard-to-guess password, or is an easy one ok? Again, it depends on what we are protecting behind this page. If we decide that the customer must use a complex password, how can we help the customer to choose this password seamlessly and point out what's wrong with any password they enter?
In other words, we need to test things out, build prototypes and talk to customers. Then do it again and again in iterations until we discover the login page that will be valuable to our customer. This looks like the design phase we discussed above, and it absolutely is.
When it's time to "mass produce" this login page that is putting it in the hands of millions of customers, we might only push a button. We deploy this page to our website, and instantly, millions of customers can access it or put it in our app and upload the binary to an app store where millions of customers can download it.
That is why "mass production" in the software industry is almost free. Design engineering (development) is about creating new things every time, while manufacturing (mass production) is about replicating the same thing all the time. Yet, many software companies today apply the Taylorism approach to their people, who should be doing development rather than mass-producing features.
Another advantage for the software industry is that we work with software, not hardware. Software is easy to change, hence the word soft. We don't need to reach a high level of confidence to deploy (mass produce) our software because we can correct things with an update or rollback, but it's impossible to update a coffee mug design while it's in the hand of the customer.
For Software, It's actually recommended to move fast and deploy often. Since this will accelerate feedback and learning, but also as important as moving fast is to practice engineering excellence, or your software will turn into hardware that is impossible to change and needs a rewrite. But let's leave this topic for another day.
The New Ways of Working
Clearly, the Taylorism approach doesn't work in the software industry, and there was no shortage of people trying to fix this. We now have agile, extreme programming (XP), Scrum, and others.
Bear in mind, this is not an attack against processes or maintaining quality or standards. This is about bringing that human element back to the way we work and treating people as they really are, knowledgable and professional.
We are human, and we solve problems and make products for other humans. We talk to our customers, discover their problems, and try to find solutions by talking to each other, debating, and challenging each other ideas to drive the best possible solutions. We deliver these solutions to customers, talk to them again to validate the solutions, re-talk to each other, and so on.
Communication is key.
When it comes to processes, and since software engineering is about the design phase rather than the production phase of engineering, lightweight processes are preferable over rigid ones. Furthermore, since the only constant is change in the software industry, we should allow anyone to challenge these processes or standards to try to come up with better ones. This is precisely what the Agile manifesto is about:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
As I wrote above, the Taylorism approach might work when it comes to the production phase, but it's not a viable approach anymore. The age of mass production has ended. Now is the age of lean manufacturing.
Pioneered by Toyota, lean manufacturing is about eliminating waste from the manufacturing process. That is, removing anything that doesn't add value as perceived by the customer. This includes overproduction and trying to forecast inventory ahead of time, and also includes internal manufacturing activities that don't add value to the customer, such as extra processes and waiting for approvals.
How do you maintain quality, then? Well, like agile, in lean, you empower people working on the assembly line to make decisions, unlike in the Taylorism approach, where workers only follow orders.
Due to the complexity of making cars and being able to compete with Toyota, now almost every car company is a lean manufacturing company. Will we see that in the software industry? Where every software company is truly an agile/lean company, or it will not be able to compete with other companies?
Toyota and other companies have achieved immense success through the implementation of Lean manufacturing, which has principles that can be applied in various contexts, such as eliminating waste and empowering people. These principles are ideas and values, not specific practices, and can be extended to development as well. However, it's essential to note that practices, like controlling inventory, are specific to manufacturing.
The book Lean Software Development adapted the lean principles nicely for software development with these seven principles:
Eliminating waste: Waste is anything that doesn't add value to the customer. This includes heavy documentation, coding more features, handoffs between teams, and waiting for approval.
Amplify learning: As discussed above, development is not manufacturing, so expect to iterate on your development and do things more than once with enough variations to experiment with things and learn what is valuable for your customers.
Decide as late as possible: You can consider this to be the opposite of waterfall development, where everything has been planned upfront. Instead, delay decisions until you have facts, not speculations, and keep your options open.
Deliver as fast as possible: Without speed, you cannot test things and have feedback. Remember, we are developing, not manufacturing. We need to discover what's valuable fast by shortening the development cycle (design, implement, feedback, improve).
Empower the team: Getting the details right is what separates excellent development from others. People on the front line, with their combined knowledge, understand the details better than anyone can make for them. So in lean, decisions are delayed to be made by people in the front line, and work is pulled by them rather than pushed to them.
Build integrity in: Software with integrity is useful, usable, maintainable, adaptable, and extensible. To score high on integrity, you need excellence in product development, design, and engineering.
See the whole: The common good will suffer if people are measured on their specialized contributions. Instead, measure, on the whole; on what makes the product successful.
Taylorism in Disguise
Applying these new ways of working without a fundamental shift in thinking is not enough. You cannot apply new ways of working with old ways of thinking.
For example, mandating an "agile framework" like Scrum, SAFe, or whatever framework is not agile. It's just like managers in the Taylorism approach, mandating how workers should do their job. This totally misses the point. Remember from the agile manifesto:
Individuals and interactions over processes and tools
There might be good teachings in these frameworks, but mandating them is the problem. Teams should have autonomy in how they work. The most important thing is to solve a real customer problem.
If a team decided that they don't want to use any process or framework and instead rely solely on communications, then why not? As long as they keep innovating and solving customer problems. If they decided to use Scrum, then as well, why not? Again, as long as they keep innovating and solving customer problems.
Focusing on frameworks had done nothing good for Nokia and Skype. Nokia lost its market to Apple, and Skype lost its market to WhatsApp. I'm sure numerous factors contributed to their loss, but one interesting thing both Nokia and Skype did was mandating Scrum.
They prescribed Scrum top-down to teams and started measuring their team success or "agility" whether they were following Scrum or not. While both Apple and WhatsApp were focusing on and measuring innovation and solving real customer problems.
I will include links about Nokia and Skype stories in the resources section below.
So, what is the fundamental shift in thinking? To relinquish control. As a leader, manager, owner, or whatever, you need to give control to capable people. A leader in the Taylorism mindset is a controller. In lean and agile, a leader is a servant leader who inspires, fosters trust, removes impediments, empowers people, aligns around common goals, and helps the team succeed.
Understanding the word development is crucial in software development. Applying manufacturing practices on what should be a development endeavor made nothing good to the software industry. Adopting lean and agile principles (ideas and values) is the way to go for any development activity. Mandating practices no matter how lean or agile a practice is, and you back to square one. Practices can be drawn from these principles taking into consideration the context around you. What is a good practice today might not be good tomorrow, or vice versa. Moreover, what is a good practice for one company might not be good for another.
Thank you for reading.