When correctness is not enough

Met the spec, but not done yet

Unknown nonfunctional requirements are among the banes of a working programmer's existence.  ("Heck," I can hear you say, "lack of any requirements...")

I'm focusing on nonfunctional requirements because it's been more typical in my experience to have the customers give a few vague verbal clues which can be generously described as functional requirements, but the nonfunctional aspects of what the customer needed are just glossed over.    Functional requirements give us some warm fuzzy feelings as programmers, because we associated them with correctness, and the correctness of an artifact w.r.t. a spec is something we can actually establish (even if imperfectly).

The usual thing people mean when they say "this software module is correct" is "given the appropriate inputs, this module produces the expected outputs."   This everyday meaning corresponds directly with the functional requirements of the module.   Alas, software may have nonfunctional requirements as well.

When software meets the user's functional requirements but fails to meet the user's nonfunctional requirements, correctness alone is not enough. [+]   The software I build can produce the right results all of the time, and still be useless if it can't run in the customer's environment, or within the time constraints the customer hasn't told me about.

It's ok to say "well, that never happens to me."  Just don't be surprised if some of your customers are reluctant to continue paying you.

When people want software to solve a problem, I think there are a couple of quantum levels of capability that "plain old non-programmer people" can imagine:

1. I have zero software to help me do this thing I want to do.  If I can do this thing at all, I do it using a manual process.
2. I have some software to help me do this task, and the speed and resource consumption of the software doesn't matter.
   
3. I have software to help me do this task, and the speed and resource consumption of the software matters greatly, because I have real-world limits on how much time or resources I can throw at my problem.

The quantum leaps happen when a customer goes from one capability level up to the next one.   The moment when "correctness is not enough" is when you have correct software which is nonetheless useless, because the software doesn't meet the user's nonfunctional requirements - e.g. it runs too slowly or consumes too many resources.

The First Leap : Basic Automation

When you (non-programmer paying customer) go from Capability Level 1 to Level 2, this is generally a Big Deal.  A manual process (tedious, boring, error-prone) is now an automated process (still may be error-filled, but likely a lot less tedious and boring for the human user).  A lot of everyday business process automation deals with this jump from Level 1 to Level 2.   These are the sorts of problems where simply having a computer being able to do the task at all, represents a big savings in cost for the customer.

Ideally, your Level 2 software process correctly does what you need it to do.   Part of the immense work required in moving from Level 1 to Level 2, is that possibly for the first time, the actual Level 1 process has to be formally described - completely, clearly, and correctly.   This formalization may appear in part as requirement documents, but from the working programmers' seat, the real deal is the code that describes exactly what the computer is going to do.
But a lot of Level 2 software customers can live with some measure of incorrectness, as long as the presented functionality still brings a net gain in economic value.   (Clearly some customers want provable correctness guarantees, and are willing to pay for them.  I think these represent a minority of paying customers.)

The Second Leap : When Resources Matter

When you go from Level 2 (resources don't matter) to Level 3 (resources do matter), this is another quantum leap.    Before now, you (paying customer) had some software that did some useful task for you, but something has changed in the resources you need to run your software.

Some possible causes -  

• you acquired lots of new customers.
  
• the acceptable level of risk associated with your software has changed.
  
• the regulatory regime changed, which means you have to produce a lot of new metadata or audit trail info that you didn't used to produce.
• the competition changed, and now has something that competes with your original automated process, but offers a better economic value to your customers.
  

Whatever the cause, now resource consumption of your software matters in ways it didn't before.  (There are different degrees of "matters" - this is not an all or nothing proposition.)   Software to solve this problem now has to (a) be sufficiently correct, and (b) operate within the allocate time or resource limits.

Software in a correctness vs constraints space

One way to look at the relationship between correctness and resource constraints, is to picture software as existing as a point in a two-dimensional space:

In this graph, points A and C represent software with minimal resource constraints, and varying needs for correctness.  Points B and D represent software with strong resource constraints (e.g. speed or memory) and varying needs for correctness.   If you are point C, and you need to be at point D, then simply being correct will not make your software actually usable by the customer.   It's a truism that "performance doesn't matter, until it does", but for some customers, when performance matters, it really matters.

[+] To say "correctness is not enough" is not saying correctness doesn't matter at all -- of course it does.  Even for customers who use somewhat incorrect software (most of us, most of the time), at some point of perceived incorrectness, a paying customer who wants your software to solve his problem, stops paying for it, because the incorrectness of the proposed solution is too expensive to live with.