Software development planning - how to avoid all the basic pitfalls and achieve your goals for once, part 1/2

So, there is some software to be created. You have a business model that would fare better with some automation and you could trim off a lot of costs in the long term or someone contracted some software development to you and you wonder how to start.

As it’s with any other things that you do, there are two basic path one might take - it’s the trial and error (or learn-it-on-your-own) approach that has its pros and cons OR the systematic approach ( find a teacher ) which usually is harder but pays off in the long run. It works for learning how to ride a bike and it works for building software.
Learning how to ride a bike on your own practically guarantees bruises on all limbs, will result in scratching off the bike paint and will take some time as you need to figure out how to maintain balance but hey! You did it on your own and owe nothing in return to the rest of the world.

It works the same way with software - just find some developers, let them know what you want in layman’s terms and send them coding.
Obviously, it will take some time to figure out what you really need instead of what you think you need and trial and error will cost you more money and part of your sanity. Not to mention relations with your counterpart. But you did it on your own.

Did it have to be like that? Could it have been done cheaper, better and sooner?
If you had someone to teach you to plan things ahead and show you all the pitfalls before you fall into? If you knew theory and practice? If you planned things?
Naturally, there isn’t any glory in doing things with help. But you co-build software that is cheaper and better and it is delivered much sooner than if you did it your way, ploughing through the pitfalls of software development.
In this article I will list the basic planning pitfalls you can fall into and at the end of it I will offer some remedies - all of those remedies are going to be… plans to have plans.
It might sound stupid at first but key problems can be anticipated and acted upon beforehand.

Why do we plan things?

Software code is complex

Unfortunately for you, coding is complex. There are few reasons why:

• Code must do some things and those things only. More complex code and bigger systems mean more points of failure and corner cases, not to mention more maintenance costs.
• Code is a strict language that requires discipline and thoroughness. Lazy coding and lack of foresight will result in reworking and more points of failure and corner cases, not to mention more maintenance costs.
• There are people involved. Not taking them seriously as a threat will result in many obscured problems appearing out of nowhere as you go.

The consequences of the above are critical. Ad nauseam - you will need specific resources ( developers and architects who know how to code and how to code well) and equipment ( computers, servers, networks).
You will need some people that know how to transform your idea of need into a workable plan for the development team so in the end it’s what you really needed, not what you wanted and within reasonable budget, time and quality. constraints
You will need a methodology of managing the software production that will result in working and reliable software
And at last and again - there are ways that work and ways to learn it on your own.

In basic terms, we make plans for software development to tame complications and complexity of it so the code is simple and works reliably. We also plan to accommodate individuals working with us - their skills and experience. And lastly - we plan to make sure that all that has been learned about software engineering and people relations is applied and we do not make the same mistakes that have been done for the past few decades building code and burning budgets.

Everyone has a finger in your cake

People involved means individuals with different skill sets and experiences. There will be attempts at realisation of their own interests during the work ( more on that later : The Individual and The Coterie ). People with experience in software development know that there are forces that will aim towards hierarchy and order within a software project and those who will push it towards chaos - one can call them good, bad, yin or yang but one has to be aware of their presence and even sometimes an overlap.
Different situations have corresponding optimal approaches and people will have their preferences about those, even knowing that it is their best interest at play, not the software’s.
Not only conscious actions are at play - in different combinations of technology and its understanding, experience of people and motivation, things can take hours or days or weeks or can’t be done in a reasonable timeframe as there are obstacles that haven’t had been anticipated beforehand.
Which means that people on your team aren’t really “on” your team in some cases.
The list goes on and the more you stare at it, the more it stares at you.
And top it off - after 50 years of “IT” and its evolution - users and customers still very rarely can formulate their requirements exhaustively and stick to them until the software is delivered.
Michael Church laid it out in one of his rants:“(...) when dealing with finicky clients who don’t know what they want, one typically has to choose between one of two options. The first is to manage the client: get expectations set, charge appropriately for rework, and maintain a relationship of equality rather than submission. The second is to accept client misbehavior (...) and orient one’s work flow around client-side dysfunction.”

Is there any way to tame this chaos? Maybe you can make some plans&work to act on those threats before they become tangible problems? Can you plan against known unknowns and the unknown?

The need to observe project constraints

On top of engineers poking on keyboards there are constraints that have to be controlled:

• Time - do you know how much time will it take until the first prototype can be shown to stakeholders? How much time until it’s done? Are those numbers rough or complete guesswork or what?
• Budget - How much can you spend, on what, is there any wiggle room for unexpected expenses. Who created the budget estimation? What does it take into account?
• Quality - What level of quality is expected? What does it mean “done”? How can we ensure optimal input from stakeholders? How do we ensure quality? How do we control it?
• Risk - What kind of risks are identified already? Are they critical? What is the procedure to manage risk in the project/product?
• Resources - Do we have people for work? Can we afford to equip them?
• Scope - What should we do? What has to be discarded?
• Stakeholders - Who’s who? Who is controlling what? How do we manage people who control things?

Lots of hard questions with uneasy answers and all leading to a conclusion that from the perspective of the product and project it is necessary to plan.
But are there other perspectives to consider?

Why people want to avoid planning things

The individual and the organisation

Yes. From the perspectives of stakeholders it might be not a good idea to plan or better not to plan at all. Matter of fact, not only it might be a good idea from the perspective of a single person in the development team/ a single stakeholder, it might be a conscious or a subconscious technique for an entire organisation or a part of it to ensure realisation of their interests. Let’s take a look at a few archetypical arguments and examine them.

1. Planning things push back the actual work

A quite simple argument that is straightforward from perspectives of an individual and organisation. It is tempting to just do it and there are reasons to do so:

• We’re not paid to sit around. Or at least in theory.
• Having the task at hand eases the mind and lets people focus on something tangible and well understood which isn’t as stressful as thinking. For example coding some code.
• People who control the money in the project/product like to see people working instead of people talking and meeting. It is not an archaic and improper approach, but skilled managers can differentiate between useless meetings and proper planning sessions.
• From the perspective of progress monitoring, it is better for the project manager (or whoever is in charge of progress) to have faster results of anything done rather than show that the team’s meeting up and discussing stuff.
• There could be some arguments with good assumptions and false conclusions that if the user doesn’t know what he wants and can’t formulate it properly why plan a failure?

Lastly, there is an argument that sticks in a situation where there is too much chaos and no solid requirements nor technology are described and the only way to go forward is to do something and see what works. Look up my article on Stacey Matrix.

2. Planning consumes a lot of time that could be invested otherwise - for example into actual work

Another good argument which makes sense from both perspectives of an individual and organisation. While, let me repeat, it is correct in chaotic and very complex environments to make “reconnaissance by combat” or in more engineering terms “let’s try a few things and see what works and burn some money meanwhile”.
However, from a managerial standpoint, planning saves time as people break down the uncertain into actionable items which are familiar to them and thus decrease the stress.

3. It costs.

If it consumes time, it costs and of course it looks better for key stakeholders when people are knuckling under rather than clocking in hours to sit in a room.
The organisation demands that things go “forward” so naturally both organisation and the individual will lean towards not wasting money on meetings but on “work” - an instinctive way to ensure self-preservation.

4. It is complicated and stressful as you try to plan things.

This is the true meaning of planning - facing complicated and stressful issues so they can be broken down into actionable items. They must be faced first and proper decisions must be made what to do about them. But it’s uncertain how long it would take to work out a solution, especially a reasonable one.

5. Planning leads to opening multiple Pandora’s boxes.

Inevitably the planning party is to stumble upon issues and matters that are complicated and remain untouched as trying to solve them will bring about conflict and resolution might not be achievable - some problems are rooted deep into people and organisation and can’t be resolved quickly or at all. Individuals and organisations know that instinctively or by experience and prefer not to touch on those topics and self preserve themselves rather than engage in disadvantaged fights - it’s called reputation management.

The coteries and alliances

They will be formed within organisations and could be cross-organisational as it is mutual interest that binds them.
Coteries in IT could be a separate article as it is a broad topic - I will focus on coteries that could be against planning and divide them by a particular reasoning against.

1. Planning will uncover incompetence

Imposter syndrome or a true incompetence, both will result in sabotaging the planning attempts. Those coteries and alliances can form among development teams, middle and upper management and usually (from my experience) root in ambiguous goals ( or lack thereof ) of organisation/ project or product.

Management’s unwillingness to sit down with the rest of the stakeholders, being unable to tackle or sometimes even name mid- and high-level problems, will not commit to a sound goal. This will trickle down onto developers/teams and cause stress and therefore result in their unwillingness to make a plan ( how can you make a plan if you do not know the goal?).

Incompetence might also be conveyed bottom-up - the working man might try to mask his own shortcomings by not providing estimations or mere plans of action because of obstacles, both real and imagined and stall planning meetings which results in bleeding eyes of managers and upper managers who try to find the root cause of the crawling chaos promising nothing except little progress and lots of costs - is it the complex environment? Is it us? Is it them? Is it all of that?

2. Planning will lead to accountability

Planning equals commitment and those who commit can be held accountable.
Sabotaging the planning due to a desire to obstruct the accountability is somewhat similar to the above but isn’t necessarily rooted in incompetence. Sometimes the aforementioned ambiguous goals or too big obstacles ( technical, organisational, environmental or personal)  could result in stalling the planning. As self-preservation in an organisation with all its benefits (compensation, prestige, options) by stalling the planning is a much more tempting perspective than pulling your socks up and getting back into the world of significance and judgement.
Such a point of view can be held by different stakeholders - development teams with little organisational empowerment to influence the product and thus feeling that accountability will lead only to pain or management teams who lack experience,knowledge or courage to gather enough influence to shape the environment in a beneficial manner to achieve a goal.

3. Planning will unveil hidden negative stakeholders

There are different reasons why someone would like other people to fail.
It could be their own incompetence (so the success of others will make me look bad) , rejection of accountability ( the conflict of own interest vs. the goal) or a more subtle sabotage tailored to a specific situation.
What would be better in that case than to stall planning and shift the blame to environmental factors and others except malicious intent?
Planning can show the disguise of negative stakeholders as they will avoid accountability, sound plans, numbers, responsibility, risk monitoring and fruitful discussions.
I have listed those reasons under coteries and alliances, not under individuals and organisations as those reasons can create coteries not aware of their own ranks - in other words, people in different levels of hierarchy or outside the organisation that work towards the same end but not being aware of each other’s intentions and existence.

The Calendar Thinking vs Let’s Occupy Our Hands With Something Thinking

To make things even worse, positive stakeholders have discrepant interests in planning per se - they do not want the same things as takeaways from those overhauled meetings we call planning sessions.
People who work for a living (in this case the developers) would like to use planning to figure out how to actually do things and how to do it “right”. They are responsible and accountable for work done well, not work done on time.
People who talk to people for a living ( in this case, managers ) would like to use planning to set milestones and control the progress as they are not responsible for work but for project and product constraints, so work is done on time and “right” things are done.
One of them doesn’t mind working on something as long as it’s needed to make it good and the other one is interested in making it good enough and shippable within agreed constraints.

Are they really in opposition? Is there a model that lets them be in harmony with each other rather than making it a tug of war throughout the entire cooperation?

Unplanned work garbage bins

All the previous combined or a combination of some (and a lot unlisted above) will result in a flood of new work-disrupting items to the list as the work progresses. This phenomenon alone is something to be expected as user tests are done and feedback is given, but if not controlled, could flip the previous work done upside down.

There are a few basic negative effects of poorly planned backlog items shovelled into workers’ desks:

• They could nullify the work already done with the new requirements and cause reworking with all consequences to project and product constraints (not to mention stress and motivation factors)
• They could lead into a never-good-enough approach where a piece of software is perfected in a stakeholder’s mind and on a test instance and not confronted with an actual user of it which also has consequences to time, schedule, budget and so on…
• They are disruptive to the team and the work schedule as a working team or whatever they can be called these days have to spend time ( relation to budget, time, schedule, risk constraints etc.) to assess, estimate and plan working on it.

From stakeholder perspectives these could be either big problems if a stakeholder is constraint oriented or can be treated simply as a cost to be taken by someone else if they are oriented to self interest only. Both types of stakeholders appear in products and projects - what can be done about the negative effects? Could it be prevented?

The problem of the complexity of software

The what do you want and the how can it be done

Building software is considered problematic as the objective of building it must be known in the beginning. A piece of software that does “something” can’t be considered useful by the user as he wants to use it to solve one of his problems - obviously.

Depending on how well you know what you want and how well you know it can be done, there are optimal and suboptimal ways to plan a product or a project. More or less agile methods to make decisions or act. More on that here.

“The science” has “agreed” on it - simply, the more you know about what you want and how it can be done - use traditional and more standardised methods and accept their advantages and shortcomings. The less you know, go towards agile methods and methodologies and accept their shortcomings.

That knowledge will never be applied globally to all products and all projects - there are two reasons behind it:

1. People are born and people die so as unpassed knowledge dies.
2. People love the idea of an easy and a stupid answer to a complex problem and therefore will always look out for a silver bullet reconciling two opposites.

‍

The let’s be Agile no matter what dogma

The trend across the globe is to lean towards agile methods as the software development cycle is shortening to ensure better software at a smaller cost.
The cycle has indeed been shortened and direct costs were indeed decreased but at a price that isn’t that easy to see at first glance:

• Kaleidoscope of buzzwords meaning something or nothing depending who’s asking. No standardised nomenclature or problems with identifying crooks using nicely sounding words.
• Tangled mess of reporting lines presented as matrix management - easy to conceal in, harder as a work environment.
• Extreme Programming and its consequences - deadlines as suggestions, scribbled requirements instead of speced out and designed workpieces.
• The “Agile” mindset - a promise of flexibility and transparency which leads to existential crisis for people looking to dress it up in any binding numbers or analytics. Or self organisation which sounds cool until you see people playing League of Legends at work.

The list goes on. Why would you like to be agile? Do you know what it means?

Is it, citing the Dilbert comic strip - no more planning and no more documentation, just start writing code and complaining?

Or is the situation really fitting the context where agile methods prevail? Do you have enough expertise and experience to be able to make products/projects like that? Planning ahead two weeks because you have no idea where the ship is headed or because there is some plan after all?

The problem of stakeholders ripping things apart

The alliances

Why do projects and products fail or are at least problematic?
Let us examine a basic matrix of key stakeholders and their expectations towards key constraints.

We can see that there are differences in expectations and they are in opposition with other constraints and with other key stakeholders:

• The User/ Client wants the product to be completed as soon as possible and as cheap as possible.
• The Contractor as a persona wants the budget extended and the deadline postponed.
• The Sponsor treats quality as a secondary goal given the fact that he’s interested in business application of the product and thus wants the product as soon as possible and as cheap as possible.

The conclusion that can be drawn from this is as follows
“ It is impossible to make everyone happy, making one stakeholder happy causes bitterness for the others. The sweet spot or a fragile balance is to make the stakeholders slightly nervous about realisation of their interests.”

Let us now take a look at a default stakeholder chart in an IT organisation that I found on the Internet somewhere.
‍

The chart shows us the default power and default interest in the IT organisation - the chart has been divided into 4 sectors and shows theoretically who to satisfy first (both upper sectors).
Below you will find a table that show default interest of all those stakeholders:

Naked eye can see a potential for alliances and coteries to form. I will list out just a few to show the perspective:

• The workers alliance - interested in life quality and professional fulfilment - might try to influence project/product in their favour by organisational politics, finding loopholes, attempts at shifting responsibility, go-slow strikes etc
• The management alliance - interested in maximising yield and business opportunities - might try to influence project/product in their favour by organisational politics, attempts at shifting responsibility
• The scope creep alliance - interested in maximising project scope and sucking out budget - will try to influence project by organisational politics, shifting blame,reprioritization efforts etc

The bigger the project/product - the greater the stakeholders environment and greater opportunities for alliance formation.
Do you have a plan to detect them and influence to your favour?

The difficult people

To rectify further, IT projects have stakeholders with difficult personalities.
Those individuals, parallelly to alliances and coteries, will try to execute their own interests within the project/product or organisation.
An attempt by Neil on Software to list out difficult personas in software projects shows us the possibilities of single stakeholders having a big impact on projects and a need to influence them on a personal level.

Quite interesting are personas with High Danger to project:

What I personally find especially interesting are Alliances ( in other name - coteries ) which have interesting consequences for project/product/organisation. I have given those coteries subjective names for quicker identification and association:

All those possible alliances&coteries, difficult personas and regular stakeholders might be problematic to detect, assess and manage.
Can it be done without planning the project?

The juggler

At the end we have the court jester - the Project Manager or a Product Manager who has to get involved in politics and find solace in the right balance of slight dissatisfaction of each stakeholder. It takes a lot of cross-domain competence to see that all of the above have their own arguments ( and therefore an interest ) in not planning ( too much ) the project.
Let us take a look:

• Pushing own changes to the product/project that are unagreed with other stakeholders
• Lack of competence in planning
• Negative stakes in the project
• Personal agenda of climbing career ladder
• General lack of competence
• Etc

Can you pinpoint certain stakeholders to each of those motives and create a plan on how to handle them?

As you can see the farther you go, the worse it gets.
Second part of the article will be full of solutions to those problems.

‍