MAY 27/Integration & Testing/4 MIN READ

Nobody Reviews the Numbers That Fly Your Spacecraft

Dan Zaidenband

Share on

Nobody Reviews the Numbers That Fly Your Spacecraft

On 30 April 1999 a Centaur upper stage tumbled out of control and left a Milstar satellite in an orbit nobody could use. The flight software was correct. One constant in the file that software loaded had been typed with the exponent one digit off, and nothing in the review chain was looking at that file.

The Centaur's control software ran a digital rate filter that called out forty constants. Most were common across missions. Five were mission-unique, entered by hand from a flight dynamics memo. On the roll filter constant the exponent went in as a zero instead of a one, so a value near minus two became a value near minus 0.2. From that point the inertial navigation unit reported roll rates at or near zero for the whole flight, whatever the vehicle was actually doing. The controller had nothing to correct against. Propellant sloshed, the stage began to tumble, and the loss is still cited as one of the most expensive launch failures in American history.

Every process in the software chain worked. The code was code that had flown before. The defect was in a file the code read, and that file was data, and data was somebody else's problem.

The schema is versioned. The values are mailed.

Open any ICD and you find message identifiers, field layouts, bit positions, units, update rates, endianness. Open the same program's parameter set and you find the numbers that decide what those fields do: control gains, filter coefficients, star tracker to body alignment matrices, thruster scale factors, telemetry conversion polynomials, mode transition thresholds, deadbands, timeouts.

The first document lives in a repository with a change history and a named approver. The second travels as a spreadsheet attachment, a memo from flight dynamics, a tab in someone's workbook, a load tape built the week before shipment. Two artifacts define the behavior at the interface. Only one of them is under configuration control.

Every integration team we've worked with versions its message definitions. Almost none of them version the load.

Parameters carry assumptions about hardware that changes

Hitomi is the cleaner case, because nobody typed anything wrong.

On 28 February 2016 JAXA uplinked new thruster control parameters to the ASTRO-H spacecraft. The satellite had deployed its extendable optical bench, its mass properties had changed, and the control parameters had to change with them. The new values did not properly account for the deployed structure.

Nothing failed for a month. On 26 March the attitude control system read a high bias rate from the inertial reference unit and concluded the spacecraft was rotating when it was not. A reaction wheel spun up against a rotation that did not exist, and the vehicle began to rotate for real. Safe hold took over and fired the thrusters. The thrusters used the parameters from February. Instead of arresting the rotation they added to it, and the spacecraft came apart.

No message changed. No interface document changed. The vehicle changed shape, and the numbers that described the vehicle to its own controller did not follow it.

We version the shape of the data and email the numbers that make it mean anything.

Your simulator ran the nominal set, not the flight set

Ask a team which parameter values their software-in-the-loop runs use. The answer is usually the model defaults, or a set somebody seeded eighteen months ago and nobody has touched since. The load that will actually fly gets assembled late, reviewed by the people who produced it, and loaded onto hardware without ever passing through a simulation.

That ordering guarantees the failure mode. The parameter set is the last artifact built and the first one nobody exercises. A simulation running on defaults tells you the software is correct. It doesn't tell you the vehicle is correct, because the vehicle is the software plus the numbers.

The fix is unglamorous. The flight load is an input to the simulation, byte for byte, and a change to the load is a reason to run the campaign again. A gain that has never been through a closed loop in simulation has not been tested. It has been reviewed, which is a different activity with a much worse record.

Parameters deserve what messages already get

A field in an ICD has a type, a unit, and a range. Apply the same three to the load. A roll rate filter constant with a declared range would have rejected a value an order of magnitude off long before the stack reached the pad. Nothing sophisticated is required. A range check is the cheapest test in aerospace and one of the least used on parameter data.

Then add the thing a message definition does not need: the configuration a value was qualified against. Hitomi's thruster parameters were right for one mass configuration and wrong for another, and nothing in the file said which. A number carries a scope. Put the scope in the file, next to the value, and a mismatch becomes a build failure rather than a safe hold event.

thruster_scale_factor_z:
  value: 0.482
  units: N/count
  range: [0.40, 0.55]
  qualified_for: mass-config-B, optical-bench-deployed
  source

thruster_scale_factor_z:
  value: 0.482
  units: N/count
  range: [0.40, 0.55]
  qualified_for: mass-config-B, optical-bench-deployed
  source

thruster_scale_factor_z:
  value: 0.482
  units: N/count
  range: [0.40, 0.55]
  qualified_for: mass-config-B, optical-bench-deployed
  source

Both properties follow from generating the load rather than authoring it. When the parameter set comes out of the same source that describes the interface, ranges and qualification scope come with it. When it comes out of a spreadsheet, they don't.

Accept that the load is part of the interface and a lot of program hygiene shifts. The parameter set gets a version and an owner. Reviews cover values, not only schemas. Simulation campaigns rerun on parameter changes, not only code changes. Vendors ship calibration data with the same rigor they apply to a datasheet. And the question at a design review stops being whether the interface is defined and starts being which numbers have flown through it.

Nobody in 1999 approved a roll rate constant of minus 0.2. It was typed, it was loaded, and no one in a position to notice ever read it again. The interface was defined perfectly. The vehicle flew on the numbers.

Share on

More from the blog