MAY 27/Integration & Testing/4 MIN READ

Everything Generated From One File Agrees. That Proves Nothing.

Dan Zaidenband

Share on

Everything Generated From One File Agrees. That Proves Nothing.

Hubble's primary mirror matched the instrument that measured it to a precision the optics industry still admires. That instrument had a lens sitting 1.3 millimetres out of position, and nobody measured it again after assembly. Every test the mirror passed was a question about whether it agreed with the same mistake.

We argue for a specific way of building interfaces: define them in code, then generate the ICD, the message parsers, the ground dictionary, the test vectors and the simulation models from that one definition. We still argue for it. It also creates a failure mode worth naming out loud, and we'd rather name it than let a customer find it.

Generation kills one class of error and leaves another untouched. Once every artifact on the program descends from one file, none of them can contradict any other. A green software-in-the-loop run tells you the flight software and the simulator read the same definition. It tells you nothing about whether that definition describes the hardware you are about to fly.

The mirror was measured against a copy of the mistake

Perkin-Elmer figured Hubble's 2.4 metre primary against a reflective null corrector, an optical instrument that shows a technician what a correctly shaped mirror looks like. Assembling it meant setting a field lens at a measured distance, using a metering rod seated in an aluminium cap. A technician seated the flat end of the rod in the cap rather than the polished rounded end. The lens ended up 1.3 millimetres from where the design put it, and after assembly nobody verified the corrector's dimensions.

The Lew Allen board found that this displacement accounted in detail for both the amount and the character of the blur in Hubble's first images. The polishing team then did excellent work. They ground the mirror until it matched the corrector, which is why the finished edge came out about 2.2 micrometres too flat, roughly one fiftieth the width of a human hair, and why it was wrong with such consistency. NASA launched in April 1990 and flew a blurred observatory until a shuttle crew installed COSTAR in December 1993.

Someone had the disagreement in hand

Two other instruments at Perkin-Elmer showed the aberration. A simpler refractive null corrector, used during rough figuring, disagreed with the reflective one. So did an inverse null corrector built to mimic the reflection from a perfect primary. Both were cruder than the reflective corrector. Both were right.

The team resolved the discrepancy by trusting the better instrument. That's the reasonable engineering move, and it discarded the only independent information anyone on the program had. Precision is a property of a single measurement. Independence is a property of where the measurement came from, and the two do not trade against each other.

Two instruments disagreeing is the only new information you get all program. Resolving it in favour of the more precise one throws that information away.

Generation turns transcription errors into model errors

On a program that keeps interfaces in documents, most defects are transcription defects. Someone retypes a scale factor into a header. Someone builds to ICD revision C while the vendor ships to revision D. Someone converts degrees to radians in the ground system, and the parser converts them again. Generation removes that whole category, and there is a lot of it.

Everything wrong in the definition itself survives. Suppose a sun sensor emits angles in hundredths of a degree and the interface file says degrees:

message: sun_sensor_angles
  fields:
    alpha: {type: int16, units: deg, scale: 1.0}   # hardware emits centidegrees
    beta:  {type: int16, units: deg, scale: 1.0

message: sun_sensor_angles
  fields:
    alpha: {type: int16, units: deg, scale: 1.0}   # hardware emits centidegrees
    beta:  {type: int16, units: deg, scale: 1.0

message: sun_sensor_angles
  fields:
    alpha: {type: int16, units: deg, scale: 1.0}   # hardware emits centidegrees
    beta:  {type: int16, units: deg, scale: 1.0

The parser generated for the flight software applies no scaling. The simulator generated from the same file emits angles that the parser reads back cleanly. The test vectors, also generated, exercise the round trip and pass. The ICD handed to the vendor prints degrees, their engineer reads a plausible number and moves on. Every check on the program sits downstream of one wrong line, so every check confirms it. The interface is off by a factor of a hundred, and the board stays green until a real sun sensor sits on a real bench.

The check has to come from outside the file

Independence comes from a test's parentage. No amount of rigour in writing the test supplies it. The useful question for a program lead is how many of your interface checks were derived from the interface definition, and after two years of tooling investment the honest answer on most programs is all of them.

Four checks that don't share the parent:

  • The hardware. Run the generated parser against a capture from the engineering model, and compare the result against the vendor's own bench readout rather than against your simulator.

  • A hand-written test. One engineer, working from the vendor's prose datasheet, who has not read your interface file. It's slow, and it's the only oracle on the program with a different author.

  • The vendor's own generated model. When their Functional Mock-up Unit and your definition disagree about a scale factor, two organisations have produced two readings of the same hardware. Log it as a finding with an owner and a date. Most programs handle it in an email thread, and it dies there.

  • Flight data from the last program. Decode an old capture with the new definition. If the numbers come out different, one of the two is wrong and you have a week to find out which.

None of this is exotic and none of it is free. It costs less than finding out during assembly, integration and test, and far less than finding out in orbit.

Accept the argument and a green integration board stops being a status. It becomes a question: green against what? A program that generates every artifact from one definition and tests only against generated artifacts has bought internal consistency and filed it as verification. Keep the single source of truth, then fund at least one check per interface whose ancestry runs somewhere else, and record which check that is beside the interface. The next engineer to open the file should be able to see the difference between an interface that was implemented and an interface that was right.

Hubble's mirror is the most precisely figured wrong surface anyone has produced. Nothing in that process failed by ordinary standards. The measurement was exact, the polishing was exact, the agreement between them was exact, and the entire chain was asking one question about itself. Your interface tooling can reach the same standard. If the only thing checking your definition is a program you generated from it, that is the standard to expect.

Share on

More from the blog