Category Archives: software quality

Where Do (Software) Architects Come From?

Where do software architects come from? There is a simple, obvious answer, but that is not what I mean :-) I am referring to company environments. I can’t help noticing that more often than not, architects come from one of the following sources:

  • Architect by appointment” (not skills). Somebody got promoted to be the architect (implying the role carries a higher value than that of a programmer), or somebody got simply pointed out
  • Not-so-great programmers who are thought to be good enough for “drawing boxes and lines on an abstract level”
  • Programmers who are so far removed from the trade they don’t have the insight any more what their options are, and what impact their decisions have
  • Non-programmers who, well, can’t code and therefore must do something that does not require coding

That is weird, assuming you choose your architect to help, improve, complement the implementation project on a level that is different from that of a programmer.

But maybe those who choose architects don’t know what to expect and what to look for. There is evidence for that: companies often don’t educate their people to be architects, or even have an expectation (that they communicate).

I can only offer my own standpoint: An architect is a person dedicated to looking at and being responsible for the quality aspects of a software project.

What kind of qualities? I would stretch that pretty far: Can the structure of the system be easily understood? Does it fit the purpose of the project? Does it address the problem and the constraints of the environment in a fitting way? Does it reduce the impact of changes to the requirements or infrastructure on the system?

While programmers must address quality issues, too, they are primarily occupied by translating functional specifications into technical reality (which is why code generation will not happen before arbitrary natural language can be understood by a computer), and wrestling with the intricacies of highly complex APIs, other people’s code, and their own.

The architect in my opinion must be a seasoned senior developer who knows what works, knows what doesn’t, and understands and can explain why that is so. S/he, too, has the (urgently) necessary people skills to promote good architecture, to convince management that additional effort will pay off, and to convince programmers that the marching direction is the right one.

You can’t study to be a “software architect”, though there are certifications. There are some great books that for sure can help. I believe a certain attitude to programming, long experience in many various projects, being awake at the IDE, and musing about why things went the way they did are a good base.

Polishing mud

[This article is a little stab at the "state of the art" as it is often encountered in software development projects]

I enjoy reading books and articles about software engineering. Particularly I value the works of Tom DeMarco and Gerald M. Weinberg. There is clearly tons of advice available on how to improve team work, and how to create better software.

But I still have to find a project that adopts more than 20% of what is purported in the publications I read.

I mean – really. You can measure cyclomatic complexity according to McCabe, apply agile methods or RUP, but as long as code contains methods named “validateOrder()” that delete stuff in the database the primary problems lie somewhere else. Where to start?

You wouldn’t polish your car before you hosed down the mud from your last SUV trip through inner Iceland. You wouldn’t use fine grit sand paper when you still have to take off a millimeter or two from a table you built. You would start with the easy things that bring a lot of effect first!

So, begin with the 20% of the effort that gives you 80% of the result:

  • Document design decisions to provide an overall picture for everybody (don’t waste your time with JavaDoc saying that “getFoo()”, well, “gets foo”)
  • Give classes, methods, parameters, and members meaningful names (that will avoid a lot of explaining via JavaDoc)
  • Keep methods and classes reasonably short (which furthers understanding, and will quite likely improve cyclomatic complexity, too)
  • Ensure that humans can easily understand what’s going on in the code (no, it is not sufficient the compiler is happy)
  • Remember: Programming is something that humans do (a lot of time is wasted when people have to reverse engineer unnecessarily difficult code)

These measures are easy to implement (especially since all IDEs support refactoring), and help a lot.

How Much Should a Developer Cost?

I recently went through the process of finding myself a new project and once more realized that at least 75% of the decision making for hiring a developer seems to be based on price. “You want – how much? Hey, I can easily find people who do it for 10 or 20% less!”

I am sure they can, for the whole “10 or 20% less”, if they just compare keywords like “Java”, “Oracle”, and “Maven”. Just as you can buy a car for $50.000, $20.000, or $5.000. While people are well aware that the price they pay has a substantial influence on the car they get, that fact seems to be ignored when it comes to hiring people.

The general assumption seem to be that once the overall skillset fits all developers are more or less the same. Well, that assumption is not supported by fact. How much does productivity vary really, you might ask? 10%, or maybe even 30%?

Tom DeMarco and Timothy Lister conducted annual public productivity surveys with over 300 organizations worldwide in their book “Peopleware” (which I strongly recommend). Here is what they found out:

  • The best people outperform the worst by a factor of 10 (that’s a whole order of magnitude)
  • The best people are about 2.5 times better than the median person
  • The better-than-median half of the people has at least twice the performance of the other half

And the numbers apply to pretty much any metric examined, from time to finish to number of defects.

Source: “Peopleware”, Tom DeMarco and Timothy Lister (Dorset House)

Wow! I am pretty sure that even a variance in salary of “only” 30% for the same job is rare, a factor of two is unheard of, and an order of magnitude is a feverish dream. And yet the performance of people you hire will probably vary by that much.

I originally had planned to continue with a long comment on what that means for the industry, for people who hire developers, and of course for the developers themselves. I am going to cut this short by simply suggesting a few things to employers and developers.

To employers

Keep the described facts in mind when you hire people. If you pay attention to quality even a larger difference in price is easily offset by substantially larger productivity – scientific research has proven that this is the case.

Think about how to interview for quality aspects. Comparing keywords or checking for certifications will not be enough, or will be even misleading. What does it mean if somebody says “I know Java”? What is their approach to programming? How would they define “good code”? How do they ensure they are a good programmer? What books do they read? What does the customer say about there work?

Regarding certifications: Certifications are mostly about being able to recall fact knowledge, and say nothing about practical experience and craftsmanship. In my personal opinion certifications mostly ensure that you looked into all corners of a specification, that you have seen the whole thing.

Taking Java as an example: It is good to know that you have Collections, Sets, Lists, and Maps in various implementations to your disposal. On the other hand, why would I need to know whether class Foo resides in package java.foo or javax.foo – it is completely sufficient that my IDE knows, or that I know where to look. That kind of knowledge doesn’t give anyone an edge over others, while practical experience in a number of different projects really does.

To developers

Highlight not only the technologies you have mastered, but also the quality of your work and your personal traits. Explain your approach to implementing a solution, what you value in good software, the books and publications you read to stay up to date and to develop your skills.

Provide some customer references that highlight your craftsmanship, that you write well structured, easy to understand, working code that comes with a high level documentation, that you are creative, innovative, professional, self-motivated, a team worker, easy to get along with.

On the Use of Tools

[Warning: This is going to be somewhat of a rant] My professional experience now comprises of 18 years (as of 2010) in companies of various sizes and industries.

Regarding the use of tools I found

  • In companies with well educated employees and good practices, the use of tools mostly increases efficiency and, to a lesser degree, quality
  • In companies with not-so-well educated employees and suboptimal practices, the use of tools doesn’t change a thing (other than now a tool is used to produce the same bad results)

And that not really is a surprise.

Example: The creation of use cases. You get good use cases when the creator has thought about

  • What is the goal of the whole thing, the purpose?
  • Who is going to read it (in the sense of “what is their role”)?
  • What information does that person need in order to do their job?”.

That is the education part. It is very helpful, too, when the company provides a standard form for use cases and explains in detail what should go into which section for what reason/ purpose. That could be called process or practice.

This all may sound trivial, but check for yourself: most use cases look like “somebody” had to fill out “some form”, just to get rid of all that empty space.

It gets especially colorful when a number of people have been writing use cases without clear guidance. Each person has their own ideas about what information goes into which section of the form.

Additionally, the level of detail and context provided varies wildly. While some will clue you in that “an order is going to be updated in the database”, the next one will jump straight into the matter and tell you to “change fields foo and bar in table baz”.

Both descriptions might lead to the same implementation, but the latter approach leaves you to reverse engineer the original intention, and robs you of the opportunity to choose a different, possibly better approach to achieve the same result.

  1. To write a good use case you don’t need a tool. Educating your employees will be your best investment, and provide the largest benefit.
  2. The use of a tool will not make a badly written use case better. See point 1. Quality won’t improve just by using a tool. What you can and should expect from a tool: to make your processes more efficient, in example by improving the organization of information you have (better overview, faster access, new views)
  3. It’s easy to write a bad use case with a tool. Again, see points 1 and 2.

You may replace “use case” with “code”, “DB design”, whatever.

Why Racing Probably Will Slow You Down

Many programmers will have gone through the following cycle.

The project time line has been tight from the beginning. Finally the specification gels into a vaguely discernible form, though pretty late. No time any more for architecture, design, and generally engaging the brain before typing. The deadline must be made! Everybody rushes to the goal, best practices and common sense are trampled into the ground, and hackers feel finally freed from those bureaucrats that call themselves architects.

Inevitably the project is late anyways, and the code is a huge mess that will be cleaned up when there is time. That is: never. Time is money; where should the money come from?

The customer? “Hey, the stuff finally works, why would we pay for beautifying the code? And, by the way, don’t you guys advertise high coding standards?”

Your own company? “Well, the budget is blown, and the customer won’t pay for it! And, by the way, aren’t you guys using best practices?”

A good solution would have taken maybe 20 or 30% longer than the ordered estimate. In the end the project consumes the same amount of time. Additionally you get a much poorer result, angry customers, frustrated programmers, and the kind of code that makes you want to switch projects before the next release. Cutting corners when your experts beg for some time to think before coding will buy you nothing in the long run.

Fighting Software Entropy – Intermediate

Recently I read an IEEE article on software metrics that identify potential dangerous constructs in code, such as brittle base classes. One short example: “Foo” declares and uses a method “bar()” that may be overridden by an extending class, potentially breaking functionality in the base class.

Metrics like this can really help to improve software quality. Considering the state of code that I see every day as part of my consulting business metrics like the one described above would be the final touch on top of a long list of measures and rules I would recommend to employ first. As long as methods with more than 200 lines abound, copying code is a design pattern, and “verifyOrder()” changes the delivery address in the data base you’ve got bigger problems to tackle first.

When washing your car you don’t start polishing before you hosed away the mud – that would be spending a lot of effort in the wrong place. Getting rid of the mud takes 20% of the cleaning effort and gets 80% of the job done. When you optimize a program you first analyze where most of the CPU time gets spent and concentrate on these hot spots. Likewise, there is a rather small number of coding guidelines that go a long way to improve code quality.

Here is my hit list of the most important code improvements:

Nomen est Omen: The name should fit the meaning. Don’t call a method “verify()” – the name strongly implies read only access – when the method changes data or even creates and deletes whole records in the data base. You don’t like documentation? Well, choose good names; together with the next rules your code will be as easy to read as pseudo code. There is no excuse for bad names: refactoring has made renaming safe and painless.

One method, one purpose: If it is difficult to find a concise name you probably packed to much functionality into one method. Break the method in smaller methods that each do only one thing, and make that thing the name of the method. Again, refactoring makes this simple.

Break down long methods and expressions. Introducing more local variables will not slow down your program; modern compilers will optimize execution better than you can (at least in this regard). Break down methods into smaller methods, too. You might even find redundant code that you can eliminate, or that methods become usable for other than the original purposes.

[This article is going to be extended]

Fighting Software Entropy – The Basics

In many companies code formatting and checking tools such as the Eclipse code formatter (or better: the cleanup function), Checkstyle, PMD, and FindBugs are pretty much the only measure taken to improve code quality. To continually use these tools is a very good measure – as long as one realizes code formatting and checking are the very basics of code improvement, not the pinnacle.

Through 16 years of professional programming experience I am certain that not tools and rules will create software quality, but genuine insight. In example, I worked for a large telco as a software architect. But instead of designing a system that, just as an example, decouples business logic from specific XML parsing frameworks through the use of domain data classes and transformers I found myself having discussions with certain programmers about the number of characters per line.

Some programmers were using expressions that extended over 200 characters and longer, and considered it their personal right and freedom to do so, as long as the code works. I disagree, since not only the compiler has to understand the code, but humans. Such an attitude neglects the rights and freedom of others, in example the right to understand the code of coworkers in a reasonable amount of time and the freedom to concentrate on implementing functionality, not having to reverse engineer cryptic source code first.

Sticking with the overy long expression example I maintain that nobody can intuitively grasp the meaning of such a monster. This is not how the human mind (specifically short term memory) works. So: Break long expressions down. Use refactoring to introduce methods and local variables. “account.isNew()” contains more information than “account.getNumber() == null” and is way easier to read and understand (and what happens if in the future accounts can be new and already have a number?).

Let us assume you are in a role that is supposed to improve software quality. Now, how do you create insight? The term “insight” implies that it is something that a person develops for her- or himself. It is nothing that you can forcefully create; it has to come from inside the person. You can try to foster insight, by appealing to having an open mind, showing good and bad examples, and providing positive feedback (eventually enjoying the results of a better programming style).

What do you do if all that does not help, and a programmer expresses his or her low opinion of the suggested programming style verbally, or worse, in code?

Well – I personally expect that professionals try to improve their skills all the time. Backed by the insight and research of others and by my own professional experience I have an opinion on what good and bad programming style looks like. I know what programming style means to a project in the short and the long term. As a developer I have often enought felt the anger, frustration, desperation, and hopelessness that comes with having to understand, maintain, and extend steaming heaps of code that nobody dares touching any more.

The difference in programming style and attitude is so great (measured by criterions such as less stress, higher productivity, more deadlines kept, lower cost, less overtime, and – yes, more joy!) that if I have the choice I’d rather work with programmers who employ a good programming style, and let the others go to work on other projects.

Mantras

[This article is going to be extended every now and then]

All Redundancy Is Evil

Really, if you can avoid redundancy, do it. If it costs some unpleasant effort, bear it – you’ll get repaid later again and again.

Copy and paste is a surefire way to pain and suffering. It is very hard to see if different but similar code really does the same thing – you’ll have to compare the two spots down to the last character. And when there are differences, are they the result of deliberate modification or some accidentally incomplete update?

If it can be generated, don’t check in the product but polish the source (and, if necessary, the generator and/ or its configuration) and automatize the process. There are way too many projects where the results of some generator run have been checked in. After a few months nobody can tell for sure anymore whether the generated stuff has been modified by hand (and don’t do that).

Don’t “Unstructure” Information

Enterprise Architect, a very good UML modelling tool, allows you to attach RTF documents to an UML element (the tech term is “linked document”). I’ve seen projects where all information is in the RTF document, and none in the fields and links that Enterprise Architect generously provides (pre- and postconditions, scenarios, actors etc.).

Why in the world anybody would do something like that is beyond me – you loose all opportunities to generate reports (ensuring a standard look and feel), run crosschecks, enforce documentation standards, perform context-specific searches, visualize dependencies, trace relationships, access information via an API and much more.

Beware of Scripting

Before I even start: I will use scripting (probably with Perl) any time to get a job done – if the job is suitable for scripting. Now you may read on :-)

I recently read an IEEE Computer Society article “In Praise of Scripting”. Having been a big Perl fan who ran into big problems with a not even sooo big Perl project I have a comment or two on the article.

In the article (and elsewhere) scripting is displayed as the method of choice for free, creative spirits who develop the groundbreaking, innovative (web) software of tomorrow while the use of compilers with strict typing is the tool of narrow-minded syntax-watchdogs who develop unremarkable applications by committee.

I beg to differ, having used Perl heavily for about seven years between 1993 and 2000. Perl does an incredible job when a single programmer has to interpret text files and transform data. This is the kind of job I still often use it for today.

But scripting languages like Perl can be a big problem when a team has to create a really large application with complex data structures and elaborate functionality. During the dot com boom a lot of very successful software was written in Perl. What people might not realize is that maintaining and extending these applications was a nightmare, and I know people who quit their job because they could not get away from having to install and maintain these contraptions.

Perl is excellent for many things including shooting your own foot. Neat features can produce obscure errors that are very hard to find. In example, variables don’t need to be declared, and have no type. Of cause it is nice that a variable just springs into existence when you need it (less typing, in example). Or that you don’t have to care about conversions from strings to numbers and the other way round. Or that complex data structures can be created, extended, and modified on the fly because associative arrays give you all that.

On the other hand, what if you misspelled a variable name? Or the second parameter of a function was supposed to be the forename of a customer, not the year of his birth? Your program will produce odd results but no error. And you then painfully realize, too, that there is no IDE such as Eclipse to help you track down the mistake you made. The compiler won’t help you either, because a = b + c is fine for it whether b and c are strings, numbers, or a combination of the two.

Some find it nice that Perl allows you to program in a number of different styles. You can make the code look like C or Java, or more like Visual Basic. There are a number of constructs that don’t exist in other languages that allow you to express yourself even more concise, such as “…unless”.

But since these constructs do not exist in other languages, people not familiar with Perl have a hard time understanding what is going on intuitively, and some constructs are just not very intuitive (“do this, this, and this – unless something is so and so” – you first follow that “this and that” will be done, but – surprise! – not under the following condition). Different styles, non-intuitive and surprising developments are great in a book or movie, but not in source code.

This article obviously is not exhausting the subject; I’ll give it a shot another time.

If It Is Awkward, It Is Probably Wrong

Just a short insight from programming an application for myself. Whenever adding or changing a feature

  • feels overly complicated
  • doesn’t quite fit into what’s already there
  • makes you feel you’d rather do something else

something probably is wrong or suboptimal with the architecture. Why? Because the opposite is so true: when the architecture is right, adding or changing features is a breeze. Just a few turns of the screw, and voila. No open heart surgery and wondering if things ever will work again like before.

As reasons why architecture turns out suboptimal, even if you develop for yourself and call all the shots, I found

  • An idea has not really been thought through. Get away from the computer, and “program” with paper and pencil for a while. Do something else for some time, and come back to tackle the problem later.
  • Changes crop up in places where they haven’t been anticipated and planned for. If an aspect of the program undergoes frequent changes, maybe consider implementing a mechanism accomodating that (a framework, plugins etc.).
  • The requirements changed so much the original approach just does not fit the problem optimally any more – acknowledge it, and change the architecture.

Just my €0.02.