The Toyota Circus and Fly By Wire

As Obama thrashes around for a way to distract American voters from his many ongoing failures, poor Toyota has found itself cast as the Villain of the Week.  It doesn't seem to make any difference that way back in 1989, the National Highway Safety Administration studied the problem and concluded that almost all cases of sudden acceleration are caused by driver error.

This should have as least calmed the issue a bit, but facts matter little when politicians need a circus to take people's minds off the situation.  Some argue that modern cars have so many electronic components and so much software that problems might be caused by the electronics in addition to driver error even though there's no direct evidence of any such issues.  The Wall Street Journal reports:

Mr. LaHood [United States Transportation Secretary] said the NHTSA will conduct a deeper review of automotive-electronics issues. "There are people that believe there are electronics problems with Toyotas, and that's the reason we are going to do a review," he said. "We don't have evidence to say conclusively that there are electronic problems."  [emphasis added]

There was the usual bickering, of course:

Several lawmakers criticized NHTSA for taking too long to act on reports of Toyota sudden acceleration problems. Some lawmakers also questioned whether the Obama administration has been more aggressive with Toyota than with domestic auto makers in which the U.S. government holds a stake, such as General Motors Co.

It's certainly true that sudden acceleration has been a problem for decades and that it's not unique to Toyota.  Let's set aside the rather obvious possibility that the government is trying to promote domestic car makers over foreign makers and trying to boost union jobs over non-union jobs.  Instead, let's stick to the engineering and consider the issue of whether there could actually be a gremlin hiding in the electronics.

Software, Software Everywhere

It's not totally bogus to suggest that the 1989 study which blamed driver error for sudden acceleration should be revisited because there's so many more electronic gadgets in cars now than there were then.

In the old days, the gas pedal pulled a cable that ran to valves in the carburetor.  When you pushed down, the valves opened.  As more air flowed into the carburetor, it mixed with more gas and the engine gave more power.  When you let up on the pedal, the valves closed, the engine got less air and fuel, and you got less power.

There were problems with sudden acceleration - when the valve stuck.  Not only was this pretty rare, it was very easy to prove after an accident: the valve would still be stuck open, jammed on engine scum or whatever, or maybe the cable would be snagged on something.  Unfortunate, yes, but no mystery, and almost totally avoidable via regular lubrication and maintenance.

In modern cars, there is no physical or even direct electrical connection between the gas pedal and the engine.  When you push the pedal, a computer is told that you want to go faster.  The computer considers the air temperature, humidity, EPA regulations, and what Al Gore had for breakfast; it may or may not command the car to go faster right away or at all.  Your input is only one of many factors the computer considers in deciding how much power the engine should generate at any given time.

You're not actually controlling the engine like you did in the old days.  At best, you send your desires by a wire to a computer which takes them into consideration as it controls the engine.  This method of controlling an automobile is called "fly by wire."

Flying By Wire, A Wing, and A Prayer

The term "fly by wire" originated in the aircraft industry.  For many years, when the pilot moved the stick or the pedals, he pulled actual cables which ran through pulleys to affect the rudder, flaps, and other control surfaces.  All these pulleys had to be kept greased, of course, or the controls would seize up.  If a cable wore out and broke, you'd generally had it.

As aircraft got bigger, pilots were simply not strong enough to move the control surfaces.  Look at the tail of a big airplane; it's the size of the proverbial barn door.  Now think about manually moving the rudder using your leg muscles when it's flying at several hundred miles per hour.

Pilots have needed power assistance since before WW II, similar to power steering in cars, but the hydraulic control lines went directly from the pilot's stick all the way out the wings and way back to the tail.

Fly-by-wire broke into the big-time with the Apollo program which landed on the moon 40 years ago.  When the rocket was being designed, the astronauts were opposed to fly-by-wire and didn't even want a computer on board.  "We got the right stuff," they said.  "Just give us the controls and we'll fly it."

Unfortunately, testing demonstrated that they couldn't fly the rocket no matter how hard they tried - human reflexes are simply not fast enough to keep a rocket on course.

Once the technology of computer-controlled fly-by-wire had been developed out of necessity, it turned out to save tons of weight on pulleys and hydraulic lines.  In the world of commercial aircraft this is invaluable; weight means fuel costs and commercial airlines can't afford to cart around the extra weight.  Having been proved in the Apollo program, fly-by-wire found its way onto commercial aircraft on economic grounds.

Why Drive by Wire?

Putting electronics in an automobile doesn't save all that much weight, and weight isn't nearly so important for a vehicle that's supposed to keep all four feet planted firmly on the ground.  Automotive drive-by-wire got its start when the EPA mandated that cars stop having bad breath and other government regulations were put in place to require better fuel mileage.

Car companies aren't stupid - they know that nobody wants to buy tiny, underpowered cars.  They'll use every trick they can find to meet EPA and other government regulations while compromising user-visible performance as little as possible.

Writing clever software to adjust the carburetor, engine, and transmission to current conditions can increase miles per gallon by 5% to 10%.  Increasing gas mileage reduces pollution, of course - the further you go per gallon, the less pollution you emit per mile.  These sorts of minute, high-speed adjustments are simply not possible to do mechanically; achieving maximum fuel economy requires complex mathematical calculations which are done in software.

Once cars had computers on board, money could be saved by putting more and more functions into fancier software as computers got cheaper.  Consider windshield washers.  When you hit the wash button, the wipers flip some number of times and the washer puts out a certain number of squirts.  It's possible to do all that with mechanical devices, but it's cheaper just to let the computer manage the washer and the wipers since you've already got a computer onboard anyway.  You can tweak the software to adjust the way the washer interacts with the wipers without changing any mechanical components.

Consider anti-lock brakes.  The first versions had small sensors which figured out that it was time to release the brake when the wheel wasn't turning as fast as the car was going.  All the "logic" was physically built into the brake drum.

Once you pay to have a computer, it's cheaper to do the calculations controlling the brakes in the same computer where everything else happens.  This reduces the number of parts in the brakes and cuts both weight and costs.

Once you've started to move functions into software, you save more money the further you go with it because the cost of putting the same software in one more car is pretty close to zero.  Every mechanical part costs something, but if you can use the same software on many different cars, the cost of "just one more" is essentially nothing.  It's hard to use the same drive train parts on a 4X4 and on a family sedan, but the windshield washer software can be virtually the same.

The Down-Side of Software Control

Having a computer involved in controlling a car or an airplane means that you've got a problem if the computer fails or the software messes up.

It's one thing if your engine is run by a computer.  It's no fun when your engine stops, but you can in theory coast to a safe stop.

Failure modes for other functions are not so benign - for example, the power steering pump stops when the engine stops and most cars are very hard to steer with the power off.  The slower the car goes, the harder it gets to steer.  With a small or frail driver, losing power steering is tanamount to having no steering at all, creating an unguided missile until the car coasts to a complete stop.

I have a friend who was driving a van-load of possessions and ran out of gas.  There was a gas station at his home exit just ahead, so he tried to coast to the pump, but he'd forgotten that his van used power-assisted braking.  He could steer with the engine off because he was moving pretty fast, but it got hairy when he started down the exit ramp and found out that the brakes weren't working as well as he expected.  He ended up literally standing on the brakes and heaving up on the steering wheel as hard as he could to get enough pedal force to slow the van so he could make the turn at the bottom of the ramp.

What if a computer failure turns off your brakes completely, or jams them on?  Or won't let you unlock the doors?  That can be even less fun than losing your engine or your steering.

Unfortunately, the computer can't see or feel what's going on; it needs sensors to know temperature, speed, fuel pressure, and any other data it needs.  If a computer or sensor failure means that your airplane can't fly at all, you're dead, as the unfortunate passengers on Air France 447 learned the hard way.

In this case, the computer itself seems to have worked properly, but ice jammed some of the instruments on the outside of the aircraft; the computer got confused and threw in the towel.  Der Spigel reports:

One alarm after another lit up the cockpit monitors. One after another, the autopilot, the automatic engine control system, and the flight computers shut themselves off. "It was like the plane was having a stroke," says Gérard Arnoux, the head of the French pilots union SPAF.

It's not unreasonable on its face for the computer to shut itself off and turn things over to the pilots when it has no idea what's going on.  Pilots get paid mostly to ride along in case they have to take over from the computer.

As it turned out, being asked to take full and unassisted control without warning in the center of a major storm proved to be a bit much.  If the pilots had been in control all along, they might have been able to handle the situation.

We may never know exactly what took down Flight 447 because the black boxes are lost at the bottom of the Atlantic, but Der Spiegel's scenario is all too plausible.  The Apollo astronauts understood these issues and were utterly opposed to fly-by-wire.  They consented only because the simulators proved that they couldn't fly the rocket themselves no matter what.

As automotive computers more and more functions such as intelligent cruise control, satellite navigation, collision avoidance, automatic parking and such, cars are coming to have as many lines of code as commercial aircraft.  That's a great deal of software indeed.

Cars driving along two-dimensional roads aren't in nearly as complex an environment as the 3-D world of aviation, but there are a lot more cars than commercial aircraft.  The more cars there are and the more software they carry, the more likely that someone, somewhere, will see the world-famous Blue Screen of Death at an awkward moment.

Software Methodologies

We've all dealt with bugs in Windows software even though Microsoft is a multi-billion dollar company which hires hordes of very smart programmers.  Most of us have cell phones; I don't know anyone who uses a cell phone who hasn't been the victim of billing "errors" which the companies blame on "the computer."

My Samsung cell phone has annoying glitches in that the alarm clock showed that it would use Ringer 1 to wake me up, but when the time came, it gave me a silent alarm instead.  A silent alarm?  Was that some kind of a programmer's joke?  Software errors are all around us.

Those of us who've wrestled with computers every day find it ironic that there is, in fact, a software development methodology which produces nearly error-free code.  The telephone companies didn't want to admit it at the time they were built, but their Electronic Switching Systems (ESS) of days gone by were really giant computers containing reams and reams of software.  They routed calls and kept track of every call for billing.

In the 1970s, how often did your phone bill have a bogus call that you didn't make?  How often did they miss a call you did make?  Virtually never.  ESS software was written using unusual software development techniques which aren't widely known outside the telephone industry.  It would be nice if they'd share their experience.

There's another way to write very reliable software which is used in automobile factories, of all places.  Auto factories are full of automatic machines controlled by Programmable Logic Controllers (PLCs) programmed in what's called "Ladder Logic."

A ladder program starts over from the beginning maybe ten times per second.  It looks at every input and computes a potentially new value for every output, then it starts over again.  The program always takes about the same amount of time to make control decisions no matter how many inputs change at the same time, and by the inherent nature of the ladder language, virtually never fails to behave as it's programmed.  Though of course it's still possible to incorrectly program the behavior, it's much easier to replicate the problem or test for potential pitfalls because the predictability is so high and the inputs, outputs, and interactions are very clearly defined.

Conventional software, such as the programs that control most aircraft, uses what are called "real-time operating systems".  A real-time OS tries to do the important tasks first and then get around to the less important jobs as time permits.

If you're in an older aircraft such as a 747, you'll note that there's a noticeable delay between the time you push the button to turn on your seat light and the time it comes on, if you push the button when the airplane is doing something that takes a lot of computing such as taking off or landing.  When it's flying straight and level, the light changes much faster.

Like the pilot's stick which isn't connected to the control surfaces, your light switch isn't connected to the light, it's connected to the computer.  The computer notices that you want the light changed and sends a command to the light to change.  Since your wants are far less important than the pilot's desires with respect to the wings and tail, however, your request gets shoved to the bottom of the "when we get around to it" list.

Avionics engineers hope that the computer has enough speed to do all the important jobs under worst-case conditions, but it's very hard to simulate the truly worst case.  Automotive software and aircraft software might be more reliable if they used the ESS or ladder logic methodologies, but those techniques are not well known outside their industries.  We may have to learn to use new methodologies the hard way.

Whither Toyota?

Our Secretary of Transportation says that the government has no evidence that there are any problems with Toyota electronics, which means that they have no evidence of problems with the software.  Toyota's boss says the same.

I believe Toyota - they'd be idiots to put cars on the road if they knew of any issues with the electronics - but that doesn't mean there aren't any.  Testing the quantity of software that's in a modern automobile is extremely expensive and fantastically difficult.  Just ask Microsoft how many bugs they're still wringing out of Internet Explorer, a very old program which is far simpler than the software in a car.

Toyota's internal memo was right to express lack of confidence in the National Highway Transportation Safety Administration, saying "the new team has less understanding of engineering issues and are primarily focused on legal issues."

Even if the bureaucrats understood the engineering issues, our government has no capability to develop or test software of any kind. Look at how well they've done at integrating the FBI and CIA databases to coordinate information about upcoming terrorist attacks!  They can't get their software act together even though our President says he's "very very angry" about their inability to share information.

For all the posturing by our elected and appointed officials, the government has about as much chance of contributing to actual safety by looking at Toyota software as you and I have of flying to the moon by flapping our arms.  That won't keep them from wasting huge sums of money looking at everybody's code, however.

Does knowing that our government plans to go over Toyota's code with a fine-tooth comb make you feel any safer?