Why Software Architects Are Essential and Why You Need One?
part 2 of 2 — The Solution
This is the second part of a two-parts article. In case you didn’t, please follow the link below to start from the beginning.
What a Software Architect is?
Talking about names and titles, a programmer, software developer or coder is whoever can write software programs. In fact, anybody can code, even children. A software engineer is much more though. She or he is a programmer, but beyond that uses a broad knowledge of computer sciences, best practices, and sound engineering principles along with his/her own experience to design, implement, test, document, deploy and maintain software. As there are many ways to do the same thing, a software architect is a software engineer, but with a broader (not necessarily deeper) knowledge in many areas, who bears the responsibility of designing and maintaining the definitions that will guide the implementation, testing, documentation, deployment and maintenance of a system or systems.
The architect isn’t a type of manager, isn’t the most senior of the senior developers, and surely is not the tech lead. The architect is first and foremost an engineer, a very experienced one. An expert in knowing a broader array of solutions in many areas (programming languages, infrastructure and cloud solutions, development methodologies and paradigms, etc). Not necessarily a specialist in each one, but just enough to reliably quantify the positive and negative impacts of adding something new. The architect also must have a far-sighted vision of how, when and where things could go wrong with the system, in order to preemptively account for them in a consistent prioritization. The architect must be an accountable stakeholder, and the stake to hold is precisely the efficiency and integrity of all the systems that he or she designs. The architect’s primary function is to lay down the rails on which code must be written, software built or purchased, artifacts registered, archived for future use, or deployed. Rails that favor convention over configuration, that define early things that would be expensive to change later. Rails that restrain the team’s freedom of creativity, yes, but in exchange for assured fluidity, predictability, and safety. Rails that keep the code clean.
The architect can’t exist in an ivory tower as I’ve often seen in some companies. The ability to do this job relies strongly in knowing exactly how it is to perform everybody else’s role, to the point of being able to switch places with anyone like an airliner captain. If one can’t know exactly how trivial or tricky some course of action really is, one can’t engage the team, anticipate risks, or assure quality.
Regarding the dev team
The architect job starts with a proof of concept, encompassing everything from benchmark research, passing through risks and costs, to final deployment. The architect should be the single or lead developer of any POC, because that is the point in time where all existing standards are revised, and all new ones devised. If the team has its own set of components and a corporate framework, that’s the time to extend and abstract details like IO devices, databases, web systems, servers, frameworks, communication protocols, cloud services, everything that needs to exist in order to support the new business cases. Although present in all stages of the development life cycle, it’s in the early stages that the architect adds the greatest value. In a sense, the development team delivers to the client, while the architect delivers to the team.
The goal of the architect is to create a shape for the system that recognizes business rules as the most essential element of the system while making the details irrelevant to that rules. This allows decisions about those details to be delayed and deferred. — Robert C. Martin, Clean Architecture
The job continues throughout all the development life cycle because we humans naturally resist to change. Team members will have doubts, concerns and possibly resentment assimilating new things. The architect needs to be there handling the transaction, fomenting the improvements, demonstrating the benefits, enriching the documentation, building trust and consensus. Sometimes something unpredictable will block the team and the architect must be ready to promptly account for the scenario. The team’s continuous fluidity is the architect’s production environment. The architect sustains on-demand maintenance requests to components, frameworks, interfaces, and documentation so the dev team is free to worry only about business issues.
The job is accomplished with the team’s engagement and enthusiasm. Good leadership skills are paramount to bring the team together around controversial decisions. Although being the one to make the strong calls on system design, the job is to guide, advocate, foment, demonstrate, and convince. Not to dictate. The team has to take part in the decision-making to secure everybody’s ownership. Fail to do so and morale will impact as much as a bad design.
Regarding the board and the clients
Last but not least is the architect’s role of stakeholder. Words from an architect should carry weight! If the team gives you an estimate longer than you expect, appeal to the architect. If those estimates can’t be squeezed to your satisfaction, assume it as your best truth. Please, remind yourself that an architect sees the feasibility of features and releases in fine grained resolution, and has no greater interest than the product itself. It’s not wise to overrule an architect’s recommendation in order to meet an imposed release date, if it ends up releasing a faulty system.
If you must press for a tighter deadline, compromises can still be made: Shorter paths now in exchange for the correct longer paths later. This is called technical debt, and it should be paid as early as possible. Just remember that everything built on top of a structure in debt can incur in more debt, and it can quickly rot your codebase to a point of no return.
Discerning architects from other engineers
At this point of the article, you might be asking yourself: Isn’t it the job description and responsibilities of a Tech Lead am I reading? The answer is no. It’s a common misconception. And although those two roles might require similar sets of skills, their focus, breadth, and depth of specialization are quite different. Aside from small companies with greenfield projects (startups), the job of architect could rarely be done by the same professional without considerable conflicts of interest.
The Architect focuses on the system design, which are determined by non-functional requirements; The tech lead focuses on the system behavior, defined by functional requirements. It’s smart to address functional and non-functional requirements separately because while the former comes from the client in a paced almost predictable manner and consistent difficulty level, the later arise from the product’s technological challenges in a less predictable way and can be quite demanding. Both of them work as leaders, but while the tech lead leads by inspiring and helping the team to do the things that should be done in a timely way, the architect leads by building trust and consensus on how things should be done.
In our railroad analogy, the architect lay the tracks in a way that the train reaches all stations the quickest without too many bumps, switches, and turns, while the tech lead drives the train making sure it arrives in all stations on time.
And it’s a full-time job! It can’t be packed in a hat occasionally wore by some other professional with other responsibilities. The architect is usually running against time to make things feasible for the team as fast as possible. When there’s no pressing requirement to deliver, the architect is upgrading the design and documentation to make things easier for the team. If it blissfully gets to a maturity level where everything is as promptly and straightforward as it can be, it’s probably time for optimization, which means upgrading or replacing whole structures.
Mind the knowledge pyramid
ThoughWorker Neal Ford blogged in 2015 about Mark Richard’s Knowledge Pyramid, which is an elegantly simple way to demonstrate in what way engineers and architects’ career long knowledge acquisition differ.
Any individual can partition knowledge in three sections:
- Stuff you know as a specialist and keep hands-on practice in order to maintain. This defines your technical depth.
- Stuff you know you don’t know, but know what kind of problems they solve, might or might not have read some papers, and can deep dive if need be, just not yet. This defines your technical breadth.
- Stuff you don’t even know you don’t know. This is everything else that exists.
These sections are not static, they shrink as time passes because knowledge in any area rapidly fades into obsolescence as the years go by, and time investment must be allocated to maintain the expertise. The things you know are also the things you must maintain.
For an engineer, and the tech lead is but the chief engineer, the main focus is to expand the top section gaining deeper and deeper knowledge on a technological niche. The deeper the engineer’s technical depth, the greater the value added to the team, regardless of technical breadth. But expanding the depth incidentally also expands the breadth, as more related technologies are encountered and superficially learned just enough to maintain the depth.
For an architect on the other hand, a broader awareness of available technologies and how it can be used to solve specific problems is more valued than a deeper knowledge of each one. It’s more beneficial to know that five solutions exist for a particular problem than to be expert in a single one. The broader the architect’s technical breadth, the greater the value added to the team, regardless of technical depth.
Types of IT Architects
There are many ways to label an IT-related architect, but all of those names change across only two spectrums: The level of abstraction and the specialization niche.
Concerning levels of abstraction, an architect starts near the code, second only to the dev team. This is the software architect, system(singular) architect or application architect. As the organization grows, more architects might be added to form an architecture team and someone from this team, usually the leader, must detach from hands-on activities to bridge the communication between this team and the business areas, dealing only with high level abstractions and maintaining a roadmap of future changes and upgrades from a business and IT process perspective. This is the enterprise architect, or systems(plural) architect. Occupying a gray area of abstraction between these two, working across all teams, high enough to participate in business discussions, but still hands-on regarding POCs of solutions, there’s the solution architect. As more organizations are turning agile, and thus getting structurally flatter, these distinctions are gradually merging.
Regarding specialization niche, there are the domain architects. Those are solution architects, close to the solution and cross projects, but they focus their technical breadth in a narrower segment of specialization. There are data architects, infrastructure architects, security architects, cloud architects, mobile architects, etc. Few organizations need each one of them, keeping only the ones that are crucial to the business. There are also what I like to call subdomain architects, like AWS architect from cloud domain or android architect from mobile domain. But really, specialize too much and you get yourself a specialist, nor a generalist, which an architect should be.
Poor Architects
Poor system designs exist not necessarily because of professional ineptness. More often than not, it’s the organization lack of understanding regarding the role of an architect that leaves the professional without options, unable to fulfill the part. I’ll anecdotally list a few of the worst practices:
- No architect at all: That’s quite common with small companies that can’t afford such an expensive professional to contribute with “nothing” to the delivery. Maybe the decision-makers really can’t grasp the necessity of an architect, maybe they are just cutting expenses in the beginning while the field is still green. The productivity of an architect-less team is bound to plummet so fast and low, dragging through so many troubleshooting and restarts from scratch, that the inflation in cost-per-release will easily surpass the cost of an architect if it were hired from the beginning in the first place. Very naive approach.
- Architect is the manager or CTO: I don’t know how or why it happened. Maybe the position of architect became a promotional rank on top of the IT structure and it’s just a name (in this case, see previous item); or maybe the team grew, and the architect just had to assume managerial responsibilities. The thing is that the role of an architect is already a full-time job, and IT management is one of the most stressful time-consuming distracting jobs there are. When someone, who is an architect and also a manager, must decide whether to take care of administrative tasks without which the whole team might be blocked, versus addressing design needs for a team full of engineers, it doesn’t take much to realize that the managerial role will always take precedence. Architecture duties will only take place when there are no pressing management concerns, which is probably never. Also, in this case, see previous item.
- Architect is the tech lead: As explained previously, those roles require different expertise focus, leadership style, and delivery objectives! The tech lead takes care of the team, while the architect takes care of the product. Whenever the tech lead/architect is pressed to deliver fast, is in fact being discouraged to engage in design activities that might get in the way. And to code without a design leads to decay. Whenever the tech lead/architect receives a great technical challenge that can’t be delegated, aside from the fact that the solution will be biased toward a certain niche, odds are he or she will submerge into lengthy coding sessions, unrelated with the user stories left to the others, leaving the team unsupervised. The team might even be senior enough to fend for themselves, but the tech lead/architect will lose control and start to miss dates. Also, because of morale, sometimes the architect needs to give the team some space for them to accept a hard change and relies on the tech lead to make a stand on the architect’s behalf, something like a good cop/bad cop routine. It’s an impractical dynamic if both are the same person. Resentment against the tech lead for an architecture decision can implode a team from within.
- Architecture committee: An architecture committee might work, in case the team or the organization is small enough. It’s a bet. If every single member is committed to putting their egos aside, reaching the best solution for the organization instead of themselves, it should work. But it’ll take precious time from projects in long meetings that raises more questions than answers, and there’ll be no documentation or fine tuning or training because every member has other priorities. The easiest thing to occur is that some members, more prominent than others, will dominate the decisions. Other easy thing to happen is that the decision won’t be the best, because once there’s no one responsible for building consensus, consensus must pre-exist, and it narrows the options.
- Out of focus architect: Also taken from Neal Ford’s blog about Mark’s Pyramid, there are architects that tries to maintain expertise in a wide variety of areas, succeeding in none of them and working themselves ragged in the process. It might even happen because the architect doesn’t understand his job, but I bet it’s the organization that doesn’t get it and assign to the architect any responsibility that doesn’t seem to have a clear owner, leaving no room for delegation.
- Frozen Caveman Antipattern: Also taken from Neal Ford’s blog about Mark’s Pyramid, there are architects who didn’t engage in hands-on activities for years and manifests a stale expertise — the mistaken sensation that outdated information is still cutting edge. Although some enjoyable skill areas will still remain, an architect is as good as his or her knowledge is updated.
- Biased architect: Architecture decisions should follow the UNPHAT principle: Understand the problem; eNumerate multiple candidates, read its Papers, determine Historical context, weight Advantages and then Think about how well each solution solves the problem. Architects aren’t allowed to worship some FAANG solutions or to decide for a technology just because it’s personally enjoyable. Everybody is allowed to be biased, except when making decisions that will impact everybody else.
- Weak or indulgent architect: As business areas will naturally be pressing for speed, the architect is the only opposing force pressing for product quality and integrity. Maybe the organization doesn’t grant the proper privileges and you have a weak architect who’s unable to perform this part of the role. Maybe you have an indulgent one, that has proper privileges but is too quick to comply with the organization urgency neglecting to measure and present the risks so as everyone gets compromised. It’s the manager job to deal with desires and expectations, an architect is also an engineer and must address the board and other areas with solid statements based on facts and evidences.
Good Architects
Architects are invisible backstage maintainers. The best way to recognize you have good ones is by the absence of the problems they solve, when things are the way they should be.
Product refinement meetings are short: Imagine the situation where your client now decides the system’s profile page must save an image along with the other text data it already does, but the respective microservice has no database or filesystem support because all the I/O is handled solely by stream. It’s not a difficult request, there are actually many ways to put it together. If you don’t have an architect, every engineer will have an idea, make proposes, a few will be singled out by the reputation of whoever put it forward and the team will discuss and draw and discuss and redraw until agreeing on an approach that isn’t necessarily the best but makes everybody happy, because everybody has to stay happy. It’ll probably end up being an amalgamation of the top approaches. And it’ll take time. Lather, rinse, repeat, and instead of a backlog refinement meeting, the topics will digress in a series of never-ending technical brainstorm meetings where little business requirements are discussed. But if there is an architect present, engineers will only have to inquire whether or not the system is prepared, in case it isn’t then when it’ll be, and proceed with business concerns. The client won’t get bored, you won’t get bored, everybody will remain engaged, the team will write all necessary tasks, it’ll come clear for everyone, job’s done, and the meeting ends swiftly and productively. That’s the way it should always be.
You know your numbers: Managers measure productivity, but architects measure indicators like how much of your code is covered by tests, is duplicated, smells bad, or has possible bugs and vulnerabilities. This is obviously not done manually, but by tools that are fit into the continuous integration/delivery process designed by the architect. At any point in time, you’ll know the level of maturity of your overall solution regarding many aspects.
Your productivity ratio is stable: In software development, productivity tends to start high and then drop as much as new features are added to the codebase. It happens because of the aforementioned code decay, since creating something new is easy but adding more features without breaking what was already working is hard. Architects guards the codebase from code decay, and if done properly, adding new features will seldom touch what is already working. Also, nobody from the team has to worry about non-functional requirements, focusing on the functional requirements which are much easier to estimate.
The team is seldom blocked by technical issues: Code rigidity or code fragility, mainly, have a way to surprise engineers with problems that couldn’t be anticipated until when actually writing the code. Then tasks become overdue or might not even be delivered. Justifications are “It was more complicated than I thought” or “I don’t know what happened, I didn’t even touch the part that broke.” Architects guard the codebase against code rigidity or code fragility, avoiding theses kind of problems.
What then?
Well, assuming you have enough years in the industry, you must have related to many or maybe all the problems presented in this article. And if I did my job well, you were able to visualize the causes behind the consequences you had to endure, the missing cog wrecking the whole machinery. Maybe you were even convinced that this is how things were with IT, and now you know it isn’t.
And if you are part of the privileged ones that get to work in an agile company, where everything runs smoothly and your worst problems are getting well-written user stories from clients, then you now know who is keeping the house of cards from falling.
If you don’t have an architect with you, do hire one with recent hands-on experience, or promote one from the most enthusiastic of your tech leads. If you don’t have enough budget to keep one in addition to a development team, don’t commit the common mistake of demanding software to an architect-less team, it’ll be unmaintainable in no time. Go with a team-less architect. It’s faster to have one release approved for productions than many release-candidates that will have to be refactored.
The only way to go fast, is to go well. — Robert C. Martin
