In need of the Software Architect

There are a myriad of potential pitfalls when selecting languages, frameworks, tools, and architectures for new projects.  I've seen so many mistakes that eat at schedule, cost, and customer expectations.  Many of the mistakes are relatively simple to fix if done early and with the right knowledge base.  These are decisions that a competent software architect is suppose to provide for your organization, group, or silo.  Only, I've seldom found a great software architect making those decisions at the various places where I have worked.  The pitfalls for picking poorly is software that is usually over budget, performs poorly, ages too quickly, difficult to maintain, never makes it to market, or multiples of these items.

What usually happens?  There seems to be a poor understanding of the position, and how it should function.  Additionally, it is mostly filled by the under qualified.  I've seen statistics that only 29% of candidates hired for architect roles fully meet the employer's requirements.  A specialist often fills the role, and then he is either incognito and almost never seen by the regular developers or he is fulfilling the role of a senior developer or team lead while also wearing the hat of the architect.  While all that is interesting, I'd like to give you my take of what a good software architect is and does instead of just ranting about the mistakes and problems in the industry.

A good software architect is a generalist, not a specialist.  He hasn't spent 20 years just coding desktop Java software, or just web stacks.  He has >15 years of total experience, but he's moved his career around a few times.  This will help shape his thinking, he won't have a narrow focus on just your industry but will have experienced wider and broader number of domains.  This will somewhat inoculate him from the development fads of the day and help him to make better decisions.  He will have at least 3 years of experience in your particular industry where you want him to architect.  This provides to him local domain knowledge and patterns of the industry with a recent historical context of where it is and where it's going.  The specialist, when hired as an architect, will have particularly good vision within his scope of understanding.  But he won't know what he doesn't know, and will tend to have false confidences about elements outside of his experience.  He will tend to leverage his own personal tools that he likes and knows well and see all problems within that context, even if that means taking a hammer to a screw.  If he realizes his limitation he will essentially outsource those aspects of his job, or if determined to do the position correctly, he will undertake the effort required to become the generalist he needs to be.  The generalist has a clearer understanding of his own limitations, he has been around specialists and he appreciates their depth of knowledge.  He has a good understanding of how various bits of a system fit together and has seen a number of technologies, stacks, languages, patterns and pitfalls that lay within them.

His role within the organization is to be that consummate generalist for the entirety of the domain in which he presides.  He will explore the latest development patterns, frameworks, and tools.  But he won't just do this himself, he will additionally task such evaluations out to the various senior engineers within the organization.  He will frequently speak with them about the warts and limitations of the current systems.  This is done individually where, hopefully, egos and politics are lower so he can get at those reservations that lurk under the surface and are seldom given voice in meetings.  He will come to know all their concerns, preferences, and ideas of where the current systems presently are and where they would like them to go.  He should also do some level of maintenance work, in some kind of organized rotation around the various products within his domain, which gives him personal experience with the current production code and a personal context for the discussions with the senior developers.  

He has access to budgets and costs for the projects so he is at least versed in the various systems' creation and maintenance costs.  He does not need to be an estimator, but that is a good skill for him to have.  He also has access to a summary of the customer feedback and satisfaction levels that he may know their view of the software systems.

Finally, where the rubber meets the road, he is the primary interface with management when it comes to sketching out new potential products.  He offers solutions that the company can fulfill, both technically and financially.  He pushes back to management on troublesome points on projects, there are often a few requirements on a new project that will absolutely drive schedule and cost to fulfill.  Maybe they are key and essential, but sometimes they are someone's half baked idea that can be jettisoned to bring schedule and costs down (something like this xkcd). Sometimes design ideas floating around in the management team are out dated or reflect inexperience, he must provide education in these places.  Basically, he protects the company from embarking on software disasters by fixing planning, expectations, at project launch.  He lowers their overall costs while improving end satisfaction, especially in the long term.