The Toyota Circus and Fly By Wire

So now bureaucrats will fix what a billion-dollar company can't?

As Obama thrashes around for a way to distract American voters from his many ongoing failures, poor Toyota has found itself cast as the Villain of the Week.  It doesn't seem to make any difference that way back in 1989, the National Highway Safety Administration studied the problem and concluded that almost all cases of sudden acceleration are caused by driver error.

This should have as least calmed the issue a bit, but facts matter little when politicians need a circus to take people's minds off the situation.  Some argue that modern cars have so many electronic components and so much software that problems might be caused by the electronics in addition to driver error even though there's no direct evidence of any such issues.  The Wall Street Journal reports:

Mr. LaHood [United States Transportation Secretary] said the NHTSA will conduct a deeper review of automotive-electronics issues. "There are people that believe there are electronics problems with Toyotas, and that's the reason we are going to do a review," he said. "We don't have evidence to say conclusively that there are electronic problems."  [emphasis added]

There was the usual bickering, of course:

Several lawmakers criticized NHTSA for taking too long to act on reports of Toyota sudden acceleration problems. Some lawmakers also questioned whether the Obama administration has been more aggressive with Toyota than with domestic auto makers in which the U.S. government holds a stake, such as General Motors Co.

It's certainly true that sudden acceleration has been a problem for decades and that it's not unique to Toyota.  Let's set aside the rather obvious possibility that the government is trying to promote domestic car makers over foreign makers and trying to boost union jobs over non-union jobs.  Instead, let's stick to the engineering and consider the issue of whether there could actually be a gremlin hiding in the electronics.

Software, Software Everywhere

It's not totally bogus to suggest that the 1989 study which blamed driver error for sudden acceleration should be revisited because there's so many more electronic gadgets in cars now than there were then.

In the old days, the gas pedal pulled a cable that ran to valves in the carburetor.  When you pushed down, the valves opened.  As more air flowed into the carburetor, it mixed with more gas and the engine gave more power.  When you let up on the pedal, the valves closed, the engine got less air and fuel, and you got less power.

There were problems with sudden acceleration - when the valve stuck.  Not only was this pretty rare, it was very easy to prove after an accident: the valve would still be stuck open, jammed on engine scum or whatever, or maybe the cable would be snagged on something.  Unfortunate, yes, but no mystery, and almost totally avoidable via regular lubrication and maintenance.

In modern cars, there is no physical or even direct electrical connection between the gas pedal and the engine.  When you push the pedal, a computer is told that you want to go faster.  The computer considers the air temperature, humidity, EPA regulations, and what Al Gore had for breakfast; it may or may not command the car to go faster right away or at all.  Your input is only one of many factors the computer considers in deciding how much power the engine should generate at any given time.

You're not actually controlling the engine like you did in the old days.  At best, you send your desires by a wire to a computer which takes them into consideration as it controls the engine.  This method of controlling an automobile is called "fly by wire."

Flying By Wire, A Wing, and A Prayer

The term "fly by wire" originated in the aircraft industry.  For many years, when the pilot moved the stick or the pedals, he pulled actual cables which ran through pulleys to affect the rudder, flaps, and other control surfaces.  All these pulleys had to be kept greased, of course, or the controls would seize up.  If a cable wore out and broke, you'd generally had it.

As aircraft got bigger, pilots were simply not strong enough to move the control surfaces.  Look at the tail of a big airplane; it's the size of the proverbial barn door.  Now think about manually moving the rudder using your leg muscles when it's flying at several hundred miles per hour.

Pilots have needed power assistance since before WW II, similar to power steering in cars, but the hydraulic control lines went directly from the pilot's stick all the way out the wings and way back to the tail.

Fly-by-wire broke into the big-time with the Apollo program which landed on the moon 40 years ago.  When the rocket was being designed, the astronauts were opposed to fly-by-wire and didn't even want a computer on board.  "We got the right stuff," they said.  "Just give us the controls and we'll fly it."

Unfortunately, testing demonstrated that they couldn't fly the rocket no matter how hard they tried - human reflexes are simply not fast enough to keep a rocket on course.

Once the technology of computer-controlled fly-by-wire had been developed out of necessity, it turned out to save tons of weight on pulleys and hydraulic lines.  In the world of commercial aircraft this is invaluable; weight means fuel costs and commercial airlines can't afford to cart around the extra weight.  Having been proved in the Apollo program, fly-by-wire found its way onto commercial aircraft on economic grounds.

Why Drive by Wire?

Putting electronics in an automobile doesn't save all that much weight, and weight isn't nearly so important for a vehicle that's supposed to keep all four feet planted firmly on the ground.  Automotive drive-by-wire got its start when the EPA mandated that cars stop having bad breath and other government regulations were put in place to require better fuel mileage.

Car companies aren't stupid - they know that nobody wants to buy tiny, underpowered cars.  They'll use every trick they can find to meet EPA and other government regulations while compromising user-visible performance as little as possible.

Writing clever software to adjust the carburetor, engine, and transmission to current conditions can increase miles per gallon by 5% to 10%.  Increasing gas mileage reduces pollution, of course - the further you go per gallon, the less pollution you emit per mile.  These sorts of minute, high-speed adjustments are simply not possible to do mechanically; achieving maximum fuel economy requires complex mathematical calculations which are done in software.

Once cars had computers on board, money could be saved by putting more and more functions into fancier software as computers got cheaper.  Consider windshield washers.  When you hit the wash button, the wipers flip some number of times and the washer puts out a certain number of squirts.  It's possible to do all that with mechanical devices, but it's cheaper just to let the computer manage the washer and the wipers since you've already got a computer onboard anyway.  You can tweak the software to adjust the way the washer interacts with the wipers without changing any mechanical components.

Consider anti-lock brakes.  The first versions had small sensors which figured out that it was time to release the brake when the wheel wasn't turning as fast as the car was going.  All the "logic" was physically built into the brake drum.

Once you pay to have a computer, it's cheaper to do the calculations controlling the brakes in the same computer where everything else happens.  This reduces the number of parts in the brakes and cuts both weight and costs.

Once you've started to move functions into software, you save more money the further you go with it because the cost of putting the same software in one more car is pretty close to zero.  Every mechanical part costs something, but if you can use the same software on many different cars, the cost of "just one more" is essentially nothing.  It's hard to use the same drive train parts on a 4X4 and on a family sedan, but the windshield washer software can be virtually the same.

The Down-Side of Software Control

Having a computer involved in controlling a car or an airplane means that you've got a problem if the computer fails or the software messes up.

It's one thing if your engine is run by a computer.  It's no fun when your engine stops, but you can in theory coast to a safe stop.

Failure modes for other functions are not so benign - for example, the power steering pump stops when the engine stops and most cars are very hard to steer with the power off.  The slower the car goes, the harder it gets to steer.  With a small or frail driver, losing power steering is tanamount to having no steering at all, creating an unguided missile until the car coasts to a complete stop.

I have a friend who was driving a van-load of possessions and ran out of gas.  There was a gas station at his home exit just ahead, so he tried to coast to the pump, but he'd forgotten that his van used power-assisted braking.  He could steer with the engine off because he was moving pretty fast, but it got hairy when he started down the exit ramp and found out that the brakes weren't working as well as he expected.  He ended up literally standing on the brakes and heaving up on the steering wheel as hard as he could to get enough pedal force to slow the van so he could make the turn at the bottom of the ramp.

What if a computer failure turns off your brakes completely, or jams them on?  Or won't let you unlock the doors?  That can be even less fun than losing your engine or your steering.

Unfortunately, the computer can't see or feel what's going on; it needs sensors to know temperature, speed, fuel pressure, and any other data it needs.  If a computer or sensor failure means that your airplane can't fly at all, you're dead, as the unfortunate passengers on Air France 447 learned the hard way.

In this case, the computer itself seems to have worked properly, but ice jammed some of the instruments on the outside of the aircraft; the computer got confused and threw in the towel.  Der Spigel reports:

One alarm after another lit up the cockpit monitors. One after another, the autopilot, the automatic engine control system, and the flight computers shut themselves off. "It was like the plane was having a stroke," says GĂ©rard Arnoux, the head of the French pilots union SPAF.

It's not unreasonable on its face for the computer to shut itself off and turn things over to the pilots when it has no idea what's going on.  Pilots get paid mostly to ride along in case they have to take over from the computer.

As it turned out, being asked to take full and unassisted control without warning in the center of a major storm proved to be a bit much.  If the pilots had been in control all along, they might have been able to handle the situation.

We may never know exactly what took down Flight 447 because the black boxes are lost at the bottom of the Atlantic, but Der Spiegel's scenario is all too plausible.  The Apollo astronauts understood these issues and were utterly opposed to fly-by-wire.  They consented only because the simulators proved that they couldn't fly the rocket themselves no matter what.

As automotive computers more and more functions such as intelligent cruise control, satellite navigation, collision avoidance, automatic parking and such, cars are coming to have as many lines of code as commercial aircraft.  That's a great deal of software indeed.

Cars driving along two-dimensional roads aren't in nearly as complex an environment as the 3-D world of aviation, but there are a lot more cars than commercial aircraft.  The more cars there are and the more software they carry, the more likely that someone, somewhere, will see the world-famous Blue Screen of Death at an awkward moment.

Software Methodologies

We've all dealt with bugs in Windows software even though Microsoft is a multi-billion dollar company which hires hordes of very smart programmers.  Most of us have cell phones; I don't know anyone who uses a cell phone who hasn't been the victim of billing "errors" which the companies blame on "the computer."

My Samsung cell phone has annoying glitches in that the alarm clock showed that it would use Ringer 1 to wake me up, but when the time came, it gave me a silent alarm instead.  A silent alarm?  Was that some kind of a programmer's joke?  Software errors are all around us.

Those of us who've wrestled with computers every day find it ironic that there is, in fact, a software development methodology which produces nearly error-free code.  The telephone companies didn't want to admit it at the time they were built, but their Electronic Switching Systems (ESS) of days gone by were really giant computers containing reams and reams of software.  They routed calls and kept track of every call for billing.

In the 1970s, how often did your phone bill have a bogus call that you didn't make?  How often did they miss a call you did make?  Virtually never.  ESS software was written using unusual software development techniques which aren't widely known outside the telephone industry.  It would be nice if they'd share their experience.

There's another way to write very reliable software which is used in automobile factories, of all places.  Auto factories are full of automatic machines controlled by Programmable Logic Controllers (PLCs) programmed in what's called "Ladder Logic."

A ladder program starts over from the beginning maybe ten times per second.  It looks at every input and computes a potentially new value for every output, then it starts over again.  The program always takes about the same amount of time to make control decisions no matter how many inputs change at the same time, and by the inherent nature of the ladder language, virtually never fails to behave as it's programmed.  Though of course it's still possible to incorrectly program the behavior, it's much easier to replicate the problem or test for potential pitfalls because the predictability is so high and the inputs, outputs, and interactions are very clearly defined.

Conventional software, such as the programs that control most aircraft, uses what are called "real-time operating systems".  A real-time OS tries to do the important tasks first and then get around to the less important jobs as time permits.

If you're in an older aircraft such as a 747, you'll note that there's a noticeable delay between the time you push the button to turn on your seat light and the time it comes on, if you push the button when the airplane is doing something that takes a lot of computing such as taking off or landing.  When it's flying straight and level, the light changes much faster.

Like the pilot's stick which isn't connected to the control surfaces, your light switch isn't connected to the light, it's connected to the computer.  The computer notices that you want the light changed and sends a command to the light to change.  Since your wants are far less important than the pilot's desires with respect to the wings and tail, however, your request gets shoved to the bottom of the "when we get around to it" list.

Avionics engineers hope that the computer has enough speed to do all the important jobs under worst-case conditions, but it's very hard to simulate the truly worst case.  Automotive software and aircraft software might be more reliable if they used the ESS or ladder logic methodologies, but those techniques are not well known outside their industries.  We may have to learn to use new methodologies the hard way.

Whither Toyota?

Our Secretary of Transportation says that the government has no evidence that there are any problems with Toyota electronics, which means that they have no evidence of problems with the software.  Toyota's boss says the same.

I believe Toyota - they'd be idiots to put cars on the road if they knew of any issues with the electronics - but that doesn't mean there aren't any.  Testing the quantity of software that's in a modern automobile is extremely expensive and fantastically difficult.  Just ask Microsoft how many bugs they're still wringing out of Internet Explorer, a very old program which is far simpler than the software in a car.

Toyota's internal memo was right to express lack of confidence in the National Highway Transportation Safety Administration, saying "the new team has less understanding of engineering issues and are primarily focused on legal issues."

Even if the bureaucrats understood the engineering issues, our government has no capability to develop or test software of any kind. Look at how well they've done at integrating the FBI and CIA databases to coordinate information about upcoming terrorist attacks!  They can't get their software act together even though our President says he's "very very angry" about their inability to share information.

For all the posturing by our elected and appointed officials, the government has about as much chance of contributing to actual safety by looking at Toyota software as you and I have of flying to the moon by flapping our arms.  That won't keep them from wasting huge sums of money looking at everybody's code, however.

Does knowing that our government plans to go over Toyota's code with a fine-tooth comb make you feel any safer?

Will Offensicht is a staff writer for and an internationally published author by a different name.  Read other articles by Will Offensicht or other articles on Bureaucracy.
Reader Comments
"Does knowing that our government plans to go over Toyota's code with a fine-tooth comb make you feel any safer?"

Of course not. Especially when I know that it will likely be reviewd by code-monkeys from India working for the lowest bidder.
March 4, 2010 10:18 AM
i think, with the brand of Toyota, which is familiar and known automobile.. it wont be difficult to get penetrate in market about this software
March 5, 2010 1:35 AM
it wont be difficult to get penetrate in market about this software
knowing that our government plans to go over Toyota's code with a fine-tooth comb dont make you feel any safe
March 5, 2010 5:27 PM
I'm currently using Toyota Avanza, and i don't have any difficulties about this issue. Perhaps because Avanza is not yet installed with the software? I'll go look about it then...
March 5, 2010 8:05 PM
This fiasco is little more than the big foot of an out-of-control government on Toyota's throat, reminding Toyota who Attila and the Witch Doctor really are. I fear much more the corruption of the most powerful government in history than I do a few possible glitches in my gas pedal. The latter might possibly kill me and a few other unlucky souls, but the former will destroy the life, liberty, and property of countless millions of unborn citizens. Now is the time to find good men, and "throw the bums out!"
March 6, 2010 12:07 AM
Relying too much on electronic device may not be a good idea. Analog device such as conventional pedal is much more reliable than this electronic device and software because most people dont understand about software and cant rewrite or debug the code. One more thing, electronic device accept many interference from the surrounding environment, and the most important is electronic device cant work without electric supplies.. isnt it?
March 11, 2010 2:42 AM
Forbes asserts that the latest episode, the 90 MPH Prius, is a fake:
March 13, 2010 1:53 PM
The VW Jetta TDI sort of symbolizes the wave of the future. 7-speed dual-clutch trannie with mag-activated dry clutch, shifts faster than hydraulic. Engine has pressure sensing glow plugs, so with that and knowing temp, you can optimize combustion over the life of the engine using a smart controller and piezo pressured injectors (faster than spring loaded) -- can get up to four squirts for each combustion event. That's a cleaner burn, which makes for less NOx and less tailpipe clean up.

Drove one. Zips through the gears keeping the engine pretty near peak torque at all times. Then you can put it in sport mode and pretend that you are shifting yourself, but human reaction times can't match those programmed in. Sort of like trying to fly an Osprey (I've not done that); can't be done without fly by wire. A human can't react to all those variables fast enough.

The Jetta TDI is fairly peppy and gets around 50 mpg on diesel, with more possibilities that can be added. That's the direction of development.

Yes, I agree that well-rounded software engineers are the key. Not enough of them, and experience does not develop overnight. If they don't know a lot more than C++, they don't understand what they are trying to do. And both hardware and software have to be tightly integrated. The old days of hanging attachments on a basic block are gone; not enough space; too much weight; and not enough potential in the package. It's sort of like aeronautics right after WWII; made bigger and bigger prop planes until they could hardly do more than lift their own weight. Had to go to jet to get enough thrust to weight ratio (although getting max fuel economy performance is much trickier than that).

Hardware and software have to develop together for these vehicles. Integrating the mess is a problem. Software from different suppliers has to work together. And yes, some of the most advanced production facilities to make this kind of stuff are coming on-line this year in China. That's where the growth is. Nobody wants to miss out. And execs will ride that low-cost machine as long as they can. That's what the financial decision system tells them to do, and executive reward systems encourage it.

Making cheap stuff will soon not even be good for China; others have gone through this cycle. Once other people understand the soft underbelly of the system, they exploit it to the max. (The U.S. got started in "manufacturing" over 200 years ago by stealing it far and square from the British -- method still works.) Seldom occurs to financially-minded execs that they are working against themselves if they are richly rewarded not to think.

Saving labor cost is usually an illusionary figment of cost accounting models -- isn't where the money is. Dumb financial models are a much bigger menace than labor cost. That's what happened to the big banks. There's a lot more to this. But that's a whole different rant.

My own emerging future vision is at
April 4, 2010 7:41 PM
The electromag interference investigation will not be confined to Toyota. When it happens EMI is unlikely to leave a smoking, which does not mean that it did not happen. Some correlations are easy to pick up. Solar flares are associated with more difficulty in cell phone communication, for instance. And remember when the solar storm (with Aurora Borealis) jiggered some control boards up at Hydro Quebec, causing widespread blackouts? They shielded the boards, and I've not heard of that happening since.

Everyone involved very deeply with accidents and safety knows that human self-reporting of what happened is not very reliable. People think they are braking, but are actually on the gas. For 2005-2010 model years, prior to media attention, NHTSA received 1113 complaints of Toyota unintended acceleration, and reports of fatal crashes with 34 fatalities where this was alleged to be a factor. Ford, in second place, had 387.

Nissan and Toyota's drive-by-wire integration did not include brake override. I think everyone else did. There's no way to guarantee that 10 million lines of code are bug free, or that some fluke has not spiked the system with a dash of EMI. All you can do is program the total system to "fail gracefully." Aircraft and spacecraft have to deal with this flying a lot closer to the sun. So NHTSA has hired some NASA electronics sleuths to review the drive-by-wire approach of the whole industry. It's not something NHTSA investigators have ever done in depth. A side note is that NHTSA funding for vehicle review and testing has been nearly flat while vehicle complexity has been growing rapidly.

Looking at this, what do you initially conclude? Some percentage of all complaints are flakey, and when looking at such stuff, it's easy to conclude that "It's no big deal." But if the outcome of a few very rare malfunctions is catastrophic, it can become a big deal, which is what happened to Toyota, and earlier to Ford and Firestone with the Explorer rollovers. Never mind that SUV drivers want to overload the things and cruise at 80 on a hot day. Before the tires blew up legally, they were considered to have great quality. Tire store returns were about 3ppm. They forgot a crucial detail: Catastrophic failures don't go back to tire stores; they go to legal impound. First discovering a problem from plaintiffs' lawyers is not pleasant.

From there on, legal logic takes over. Who know what, when, and what did they do about it? Low percentages mean nothing. Legal cases are about each catastrophic outcome. If internal communications suggest that a company knew it had a serious problem and willfully did nothing, it had as well settle up immediately. The most business-friendly jury in the country is unlikely to find for the company. (That was the situation with Ford and the Pinto gas tank.)

The moral of the story is that it's not 1930 any more. Then the first car radios were ungrounded; could not filter out ignition static noise when grounded. So a lot of cars caught fire, including that of the President of Motorola. From Motorola history, nobody sued. Any radio beat hell out of no radio. Today, the customer expects the highest of tech to be served up to perfection. Nobody, including auto designers, can explain in detail how everything in one of these beasts works, but they expect "the system" to deliver perfection, or very close to it.

April 5, 2010 5:31 PM
I relied on NHTSA figures compiled by Edmunds at:

Edmunds analysts are pretty good. Their business depends on them.

I'd like incidents per unit too, but don't have time and money to get it. You can only get vehicle population estimates from state registration data compiled by Ward's Automotive or by the Motor Vehicle Manufacturers Association; their databases cost money. I'm guessing that Toyota put more drive-by-wire vehicles on US roads during the past six years, but not by a lot. Ford sold more units in total (including trucks that weren't drive by wire), but the drive-by-wire population probably is not materially different. To be precise, you have to decipher which models were drive-by-wire brakes, not just ABS, and when.

Mercedes recalled about 600,000 of their E-Class in 2005 with an embarrassing glitch with their first electronic brakes -- but they recognized that software was at issue and did not get such a black eye over it. Should have been more of a warning to the industry.

For practical purposes, my SWAG is that Ford and Toyota vehicle populations on the road were "in the same ball park." Toyota had a clear spike in complaints, whatever the cause. The NHTSA database is a bear to interpret, so I relied on the NYT reporter's summaries at:

One of the best sites I know of in this, with carheads revving on the Toyota problem, is's-wrong-with-toyota's-black-boxes/

IEEE has a good blog or two also. Minimal political BS; and focused on the tech stuff.

But now we're out of operational and engineering logic in the Toyota mess, and into legal logic. From a plaintiff's view, big number data is only background pointing to culpability. One must establish intent to neglect, deceive, or cover up. You'll note that a lot of the electronic blogging is on Toyota possessiveness of black box data after crashes. If a lawyer "discovers" that Toyota willfully withheld key, damaging post-crash data from either NHTSA or the plaintiff attorneys, the black eye is going to spread all over Toyota's face.

Yes, Toyota is probably still #1 in replicative quality in plants, but that's not what this scandal is about.
April 6, 2010 6:02 PM
Quotation of the Day, NYT, 4/6/10

"We now have proof that Toyota failed to live up to its legal obligations. Worse yet, they knowingly hid a dangerous defect for months from U.S. officials and did not take action to protect millions of drivers and their families."
- TRANSPORTATION SECRETARY RAY LAHOOD, announcing that the department will seek to fine the automaker $16.4 million.

And the blame game goes on.....
April 6, 2010 6:11 PM
Pretty good piece putting the Toyota acceleration problem in perspective by the NYT this morning.
It's at:

NHTSA data is hard to plow through and they don't summarize it in any convenient spot I've found. Go to the case by case reports they do put up, and it's obvious why interpretation is dicey. You have to decide from the complaint whether something unintended is going on with the vehicle, or whether the complaint is from someone with unreasonable expectations or ignorant of how vehicles work. If you work this stuff all the time, it's pretty easy to go to sleep -- dismiss every compliant as just another crank rant.

The best companies rotate people in customer complaint positions, and they let a lot of employees listen in on customer service calls. It gives them a better perspective of who they really serve and what's important about their job. Listen to complaints all the time, and you get jaded. The phrase "The customer is always right" is from J.C. Penny, originating long before Penny went into the car service business.

Speculatively, something else may be at work here -- psychology of control. When I first began to drive, I changed a flat once a week or so (could afford neither new car nor new tires). Took 10 minutes max; no problem; it's how life was, and I could fix it myself. Today, you need special tools and maybe roadside service to fix a flat. That means that if I have a flat, I'm dependent on somebody else, and it's cost will not be 10 minutes work, plus another patch on an inner tube. Most anyone in the 1950s could keep a car running with a modicum of mechanical aptitude and a basic toolbox.

But now we've almost come to where we're dependent on the special tools and diagnostics of the OEM to keep a car running. In effect, the business model is transforming into a life cycle service business. If you can't fix it yourself and don't understand what the hell is happening, the level of vehicle performance and service in the field better be very good.
April 9, 2010 6:38 PM
To put things in perspective:

Speed Control a Small Factor in Car Claims
The number of claims involving speed control was small compared to reports that listed air bags as a problem.

Remember, air bags were required by law. Car companies had offered them, but most customers preferred to use seat belts which were somewhat cheaper. Then the government decided to force the customers to pay for the things. It costs billions per year for air bags which save maybe 400 people. A nutty way to spend money.
April 9, 2010 6:45 PM

Early U.S. Tests Find No Toyota Flaw in Electronics
Findings by a federal safety agency support Toyota's claim that it had identified and was fixing the reasons that its vehicles might accelerate out of control.
August 12, 2010 5:24 PM
Add Your Comment...
4000 characters remaining
Loading question...