Wednesday, October 30, 2013

Unintended Acceleration, Software, and Sadness

A few years ago I became concerned about reports of sudden unintended acceleration in Toyota vehicles, especially when some of my family members started driving new Toyotas. At first I was skeptical of the reports, but they kept coming. In time, a friend of a friend had a terrible accident, and I was only two trustworthy people removed from a firsthand experience.

I started paying more attention to the reports, and I developed a very strong suspicion that software was to blame. Three simple facts led me to this suspicion:

  1. The engine throttle was controlled by software.
  2. The brakes were controlled by software.
  3. Nobody knows how to make software without bugs.

The first fact surprised me a little. The second surprised me a lot. The third is common knowledge to anyone who has ever developed software, but it may be surprising to those who haven't.

When I first learned how to program in BASIC as a child, I was taught that computers don't make errors; people do. If you write a perfect program, the computer will do exactly what you expect. This is a tantalizingly optimistic view, and it helped me challenge myself to become a better programmer. Unfortunately it is not true.

During those early years my programs were small and simple. Sometimes I could write an entire program in a single page of text. It seemed very possible that programs could be perfectly correct, but somehow they never were. There was always a bug, and almost every time the bug was my own fault. As time went on, I started working on larger and larger software projects, and it became clear to me, as it should to any software developer, that the likelihood of bugs increases when software complexity increases.

This is a Big Problem. It is so big that many of the greatest minds in computer science have devoted their lives to it. Some very interesting progress has been made, but it is still largely an unsolved problem in real-world systems. It is hard to create a program that correctly implements a specification. It is hard to create a correct specification. It is hard to implement a programming language correctly. It is hard to build correct interfaces to other programs. It is hard to build computers that reliably execute programs correctly, especially in environments with high levels of electrical noise like the engine compartment of a car.

By the way, I often use "is hard" to mean "might be impossible".

Not all software is equally buggy, of course. It is possible to create computer systems that are more reliable than others (consider the hardware and software on spacecraft, for example), but it is difficult to do. It is a very different problem than the problem of building reliable mechanical systems.

The problem of software bugs is probably the biggest reason that computer security is so awful. We don't know how to make software without bugs, and bugs tend to undermine security. This is why people in the information security community seem to understand and expect bugs more than most other people; we spend our lives discovering, analyzing, exploiting, and fixing bugs. We find bugs that others miss. We break things that are supposedly unbreakable.

To me, the unintended acceleration reports smelled like buggy software from the very beginning. Few of the reports were identical, but all of them involved the inability of the driver to influence a computer that controls the engine throttle.

Some of the reports agreed on a particular point: Pressing harder on the brake pedal did nothing. This is terrifying to imagine. Your car accelerates rapidly even while your foot is on the brake pedal. You press harder and harder until the pedal is at the floor. Maybe you have time to switch off the ignition or shift into neutral, but how long would it take you to think of that? It might take only a second of unintended acceleration to cause a fatal accident.

At first Toyota denied the problem. Then they recalled floor mats. At the time, I thought that was a pretty stupid response to what seemed like a software bug. Then they recalled pedals. Then they blamed the drivers. They repeatedly said that they couldn't recreate the problem when testing the software (but any software developer knows that an inability to reproduce an error rarely means that a bug doesn't exist).

I started wondering: Have any information security professionals audited the software? Has anyone actually skilled at finding bugs looked for bugs? As far as I could determine, the only people who had tested the software were automotive engineers employed by Toyota. Automotive engineers might not know anything about finding bugs, but they should at least know something about fail-safe design.

To me, the most troubling part of the whole thing was that the brakes and all fail-safe mechanisms were also under computer control. Really? You would make a car with software throttle and also give it software brakes? Don't you know that an automobile is a lethal weapon? Have you never seen software fail? How about a traditional brake system just in case, even if it is only activated when the brake pedal is fully depressed? How about a mechanical linkage that limits the throttle when the driver slams on the brakes?

I can't imagine any engineering culture within Toyota that would fail to consider such things unless it is simply a case of automotive engineers putting too much trust in software because they don't understand software failures. Maybe they tested the things ten thousand times, unaware that they should have tested ten trillion different conditions.

As I became more and more convinced that a software bug was to blame and that nobody was properly looking for it, I started planning a blog post. I considered trying to reverse engineer a car. Even better, perhaps I could convince someone more skilled than me to try to find the bug.

Then the unexpected happened: Tin whiskers were implicated as a cause of unintended acceleration in Toyota vehicles. I had convinced myself that software must be to blame, but suddenly a seemingly plausible alternative arose. I understood tin whiskers well enough to believe that they could explain at least a portion of the failures, yet tin whiskers were just mysterious enough that I didn't question whether or not they might explain all of the failures.

Then I failed. I stopped paying attention after I heard about the tin whiskers. I didn't consider the likelihood of software bugs vs. failures due to tin whiskers. I didn't follow through on making recommendations for mechanical fail-safe (which could prevent fatal accidents regardless of the root cause of the problem). I didn't notice when Toyota denied that tin whiskers caused unintended acceleration. I never went back and reviewed the notably weak software analysis results of the NASA report that first implicated tin whiskers. I ignored the fact that the United States government stopped investigating the problem.

This week I read that a court of law found Toyota's faulty software to blame in a case of unintended acceleration. A software audit for the plaintiff revealed that coding standards for safety-critical software were not followed and that the software is buggy and incredibly complex. The audit even identified a particular failure mode in which a driver could press harder on the brake pedal with no effect, which is as close to a "smoking gun" as we could hope to see. The case clearly indicates negligent software development and deployment practices on the part of Toyota.

This shouldn't have happened if the automotive engineers were appropriately skeptical of software. This shouldn't have happened if the executives were appropriately skeptical of software. This shouldn't have happened if the software engineers were appropriately skeptical of software.

At the very least, the software engineers should have known better. If I were developing software that could kill someone in an error condition, I would feel a moral obligation to tell people about the potential for error. However, as everyone in the information security community knows, developers tend to overestimate the quality of their own code, and very few software developers are skilled bug hunters.

Unfortunately the software source code still has not been made available to the public. We have to trust the analysis of the plaintiff's expert witness (or trust Toyota) to understand how the software works. The details from the expert witness that have been reported, however, seem very credible to me. The jury found in favor of the plaintiff, so Toyota failed to effectively argue against the analysis.

I'm pretty confident in agreeing with the analysis, but it would be nice to be able to verify. If the software were open source, that would be possible. In fact, if the software were open source, others could have done the same analysis years ago and likely would have been able to fix bugs and save lives. How many people will have to die before we decide that open source is as important for safety as seat belts?

I am deeply sad for the people who died in automobile accidents for years before Toyota's negligence was revealed, I am sad for the people who will die in future accidents, and I am sad and ashamed that I never followed through on my own suspicions about the bugs at the heart of the problem.

58 comments:

Redbeard said...

My favorite bit about this from the past:

“The jury is back, The verdict is in. There is no electronic-based cause for unintended high-speed acceleration in Toyotas. Period.” - Ray LaHood, United States Secretary of Transportation

Anonymous said...


Modern cars need 'flight data recorders' for diagnostics and investigation of anomalous crashes and near-misses.. it's not even hard... 'candump' on the multiple CANBUS's in the vehicle.


.... *Therac-25* ....anyone?! .... “Those who cannot remember the past are condemned to repeat it.”

Perry E. Metzger said...

One note: formal methods are no longer beyond usability for this sort of thing. We now have a formally verified C compiler (CompCert), a formally verified microkernel (seL4), and other similar gadgets. Ten or twenty years ago formal verification of large systems was inconceivable, but now it is actually quite doable thanks to massive progress in the technologies to assist people in doing the proofs.

Anonymous said...

Assuming the accelerator and brake systems run on CPUs built on something like 1 square centimeter of silicon, in aggregate Toyotas could represent one of the world's larger cosmic ray detectors.

- Marsh

Anonymous said...

Never underestimate the power of an expert witness to uncover the truth. It must be a very rewarding job indeed.

Robert Graham said...

The number one cause of unintended accelerator is because drivers panic and press the gas pedal instead of the break. It's a constant across all manufacturers and all models. It's not even clear that the problem was statistically more prevalent in Toyotas.

Unknown said...

Michael, I can almost feel the conviction with which you wrote this post. What stood out to me was your statement about how open source can greatly enhance safety and security. I never thought of it that way.

Thanks for the great post. And by the way, waiting intently for HackRF general shipment :)

Omer Ansari

your MASTER said...

anyone know anything about experimental remote access interface devices? I seem to be finding evidence there is an exper4imental Bluetooth device accessing my systems- I was havng serious invasions through a router and modem owned owned by Michael shurer, owner of 'MICHAEL'S AUDIO AND VIDEO' in fraser, co.- he also owned an apartment building and I was a renter ther, and since june of last year, when I saw the "?" between my pc's and his router,apparently I have 3 interfaces ands on wireshark, I was shown indication of an experimental device-REALLY! sincwe I moved from sharers apartment bilding, the crap has gotten worse, I fight for use of my device every dfamn time I try to use it. the fact that I had caught invasion from employees of michaels audo and video of my satellite dish should be reason for an investigation, and I am trying to figure all the ways these devices are being accessed and I find experimental bluetooth devices? I even found files for using my speakers as microphones and adding files remotely- and it just gets worse from there... anyone with any info as to why I am being shown these things in scans, call 970-887-9695 and help me find these damn hackers- they gotta be within Bluetooth

your MASTER said...

now-I just gotta find this thing in my home- any ideas as to the distance the hackers have to be to access a Bluetooth device?

your MASTER said...

now-I just gotta find this thing in my home- any ideas as to the distance the hackers have to be to access a Bluetooth device?

asor said...
This comment has been removed by the author.
Raymond said...

This is quite unfortunate. You're right about the idea of creating bug-free programs, moreso on complex ones, is next to impossible. You'd imagine they would have tested the software, then did a thorough test drive and analysis, before putting it on the market. Several layers of testing, both in the software and the hardware side of things, could've helped in avoiding issues like this. Thanks for pointing these out!

Matt Wynan @ Innovative Defense Technologies

Highly recommended Fishing Lodge Alaska said...

electrical noise like the engine compartment of a car.

Highly recommended Best Maid Service Austin Reliable Dependable said...

concerned about reports of sudden unintended acceleration in Toyota vehicles,

Highly recommended The FL Orlando Wedding Venues said...

The engine throttle was controlled by software. - yes indeed it's kinda scary, manual driving is more safe than this.

Unknown said...

this is a right information. unintended acceleration and software is very useful post. thanks for sharing.
full version software sites

patricianapoleon said...

very nice sharing! this is really a good information of Unintended Acceleration.
crack download manager free download

Unknown said...

internet secure software is really useful for pc and thanks for informing.
crack trial software full version

Unknown said...

it is very useful post! thanks for giving nice information.
cracked software for pc | download software crack

Unknown said...

that's really awesome...thanks for giving informative post.
android data recovery pro full

Agen Bola Terpercaya said...

woow nice topic Agen bola terpercaya

Agen Bola said...

This is the kind of manual that needs to be given and not the random misinformation that's at the other blogs. Appreciate your sharing this best posting.
Agen Bola

Unknown said...

thanks a bundle....
Microsoft Outlook 2016 For Mac Free Download Full Version | Soda Pdf Crack

"1A Advanced Garage Doors " said...

I'm always interested in software discussions.

Unknown said...

Well, it’s a nice one, I have been looking for. Thanks for sharing such informative stuff.
toyota vehicles

Star said...
This comment has been removed by the author.
jade said...

It is the best for planetary astrophotography, but Idon't know which format does Stellacam EX control software use to save video files. field service software 

Commercial Equipment said...

Recognizing USB hyperlinks is often difficult for your handful of nonetheless it’s straightforward in case you associate with them constantly. You'll need to differentiate USB hyperlink connectors from Form A, Kind B or Mini-B. USB ports about the PC are dependably USB Variety A ports. The dimensions could have USB Style B port, so all you demand is often a connector that may have both equally toward the tip And so the machines can with no much of a extend be associated with one another to operate legitimately. Advantages Of the dimensions application

agen judi maxbet said...

Maxbet adalah permainan yang dapat memainkan taruhan Esport, bagi kalian yang sangat suka bermain permainan esports tentunya kalian ingin mendapatkan jajanan tambahan bukan ? silahkan kunjungi situs kami dan dapatkan uang jajan tambahan untuk kalian.
http://www.agenmaxbett.com/daftar-judi-bola-maxbet/

http://www.agenmaxbett.com/deposit-judi-bola-via-pulsa/

http://www.agenmaxbett.com/aplikasi-maxbet338-android/

http://www.agenmaxbett.com/agen-maxbet-terbaik-dan-terpercaya/

Zanna said...

PERMAINAN ONLINE TERBESAR DI INDONESIA

Website paling ternama dan paling terpercaya di Asia ^^
Sistem pelayanan 24 Jam Non-Stop bersama dengan CS Berpengalaman respon tercepat :)
Memiliki 9 Jenis game yang sangat digemari oleh seluruh peminat poker / domino

- Adu Q
- Bandar Q
- Bandar Sakong
- Bandar Poker a
- Poker
- Domino 99
- Capsa Susun
- BANDAR66
- PERANG BACCARAT (Games Terbaru)

Permainan Judi online yang menggunakan uang asli dan mendapatkan uang asli ^^

* Minimal Deposit : 20.000
* Minimal Withdraw : 20.000
* Deposit dan Withdraw 24 jam Non stop ( Kecuali Bank offline / gangguan )
* Bonus REFFERAL 15 % Seumur hidup tanpa syarata
* Bonus ROLLINGAN 0.3 % Dibagikan 5 hari 1 kali
* Proses Deposit & Withdraw PALING CEPAT
* Sistem keamanan Terbaru & Terjamin
* Poker Online Terpercayaa
* Live chat yang Responsive
* Mensupport 8 jenis bank lokal dan tersedia deposit via OVO dan PULSA TELKOMSEL serta XL


Contact Us
Website SahabatQQ
WA 1 : +85515769793
WA 2 : +855972076840
Telegram 1 :+85515769793
Telegram 2 : +855972076840
LINE : SAHABATQQ
FACEBOOK : SahabatQQ Reborn
daftar sahabatqq

#sahabatQQ
#winsahabatQQ
#winsahabat
#megasahabatQQ
#rajakartu99
#windaftar

Field engineer said...

hi

Field engineer said...

Hi,

Obstacles Hindering the 5G Smooth Deployment

5G represents exciting opportunities for our businesses and domestic lives alike. It could help us perform tasks that would generally cause huge bandwidth limitations with ease. It could unlock exciting possibilities in terms of virtualization and the cloud. However, its deployment is proving to be a bumpy road.5G will require great innovation and renovation in terms of architecture and platforms. Central networks and radio access networks must be fully retrofitted or replaced.

Anonymous said...

nice post
executive protection
Best Way to Update Windows Drivers

office.com/setup said...

Hоw dо ореn-ѕоurсе рrоduсtіvіtу ѕuіtеѕ compare tо MS Office - аnd dоеѕ іt mаkе ѕеnѕе fоr уоur оrgаnіzаtіоn tо сhооѕе frее соmmunіtу software rаthеr thаn Microsoft's commercially licensed оffеrіng? www.office.com/setup www.office.com/setup

NOC Performance from Servers said...

Managed Network Services gives you access to a wider pool of experts to help manage your growing business network needs. It provides proactive support to ensure issues are fixed in a timely manner and offers improved IT security to help protect your customers and employees. It also simplifies your cost management, resulting in a more productive and cost-effective business.

ALA HOSTELS said...

Impressive!Thanks for the post
You’re doing a great job Man, Keep it up.
Kexlin provides quality and niche candidates who will support the strategic objectives of the organization.
IT Consultancy in Hyderabad
IT Consultancy Services in Madhapur Hyderabad
Software Consultancy Companies in Hyderabad
IT Service Based Companies in Hyderabad

Amazon my tv said...

Through www.amazon.com/mytv - how you can connect your mobile phone to Amazon Prime. Through amazon.com/mytv, you can watch your favorite TV shows, series movies. You can watch prime videos anywhere on your device. Users need to create an Amazon account if they don’t have an Amazon account and enter the Amazon my TV activation code to watch Amazon prime videos on your device.

amazon.com/mytv | www.amazon.com/mytv

Brooklyn Smith said...

Get the help of professional writers to write your diploma assignments & project. GoAssignmentHelp.com writes assignment for accounting, law, building & construction assignment, finance, economics, programming assignment help and etc with best prices. Most importantly our experts have earned their degrees their major topics from the best Universities of US, Australia or UK. we provide you a unique and plagiarism free content every time you order an economics homework help, GoAssignmentHelp.com always try to deliver the assignments before the expected deadline moreover we have a free rework policy that too for the lifetime. You are warm welcome to GoAssignmentHelp.com so call us without any hesitation and get the more information about case study help. We are with here for your help.

Jack Stark said...

Sankey Diagram is the best visualization to improve your SEO. Sankey diagram is a very useful visualization to show the flow of data.ChartExpo provides you a better and easiest way to create the Sankey Diagram in no time without coding only on few clicks. Sankey Diagram.

david anderson said...

Look no further for Assignment Help in Canada, as we have experienced professionals who can craft your content in no time. We deliver authentic assignments that are written from scratch by gathering relevant information from reliable sources.

Unknown said...

I'm Khloé Zac, I tried to invest my savings into forex broker's trade during Pandemic and ever since last year December have been trying to withdraw my savings and each time i try to withdraw i'm asked to pay for fees and Tax fees, last Month june 26th i discovered that it was all scam and i have already lost $450,000 US dollar's. I was referred by my bestie who know so much about the internet and he referred me to Vitor programmer, i emailed him and he asked me to get in touch on WhatsApp and i did as he instructed, after 32 hours of reaching out to Vitor Programmer i received a notification on my phone screen and it was blockchain and my funds were recovered full without stories, i am writing this because a lot of people complain of being scammed online while trying to have there funds recovered, kindly Email: Vitor@programmer.net, WhatsApp contact: (+1) 519 / 398 / 1460, and tell him that you are from Khloé Zac.

assignment help said...

Let’s think of a situation, you are very good at studies and serious about your grades, but this one take my online class is creating a toll on your plans. You have already tried your best but all your efforts are getting into vain. Now, you are standing at the verge of online assignment help submission and chances are that you will lose the desired grade. What will you do? You will ask for help, right? This is where the role of assignment help comes into the picture. They help students when there is no way left to secure a decent grade. And, there is nothing bad in asking for help in need.

assignment help said...

Deadlines are often scary. Aren’t they? If you the same then you are in need of someone who can wipe this fear away. And professional assignment help can do this thing in such a smooth way that you will not even realize. All you have to do is to carry out a little bit of research on the programming assignment help web.

Jimmy Walter said...

If you want to progress in academic writing, you need to count every single step. Ensure to follow the right path and add essential qualities to your writing curve. Asking the Assignment Help of someone to write my assignment may sound instant solution but it can help you to boost your learning exposure. Write my assignment | Homework Help | Accounting Assignment Help

noaholiver said...

It is not mandatory to get help from experts in college. There are several reasons why students approach experts and say I need to pay someone to my homework done online . If you are also planning to do the same, you might want to look at some common reasons why students need expert assistance. If you are interested then here you go. When deadlines are too short to be met, students often get panicked and they rush to the nearest helpers for academic assistance. When the homework questions are too tough to solve and students lack the fundamental knowledge about questions, they need help.

ceaaviation0 said...

CEA Aviation is located in the state of Delhi, India. It is affiliated to the "Top College for Pilot Training in Delhi" Another hostel is available here with many facilities. There are also additional campus facilities like Math Lab, Computer Lab, Counselling, Fest, Cultural Zone. What facilities do you want to take in this college? All facilities are here. As the.

Anonymous said...

CEA Aviation is located in the state of Delhi, India. It is affiliated to the "Top College for Pilot Training in Delhi" Another hostel is available here with many facilities. There are also additional campus facilities like Math Lab, Computer Lab, Counselling, Fest, Cultural Zone. What facilities do you want to take in this college? All facilities are here. As the Best Commercial Pilot Training Program in Canada.

reword essay said...

In order to write something like experts do, you will have to work hard. Essay helpers are generally professionals who wholeheartedly help students with their essay writing tasks. With their assistance a lot of students gain good grades and learn valuable essay writing tips as well. If you are looking ahead to write an essay like experts, you can monitor their work closely. Go through some expert-written essays closely and try to imitate their style for a while reword essay. You can also get in touch with essay helpers and ask for suggestions. They will surely help you by sharing some simple yet important essay writing tips.

Assignment Help CA said...

Studying in New Zealand forces you to take care of your assignments and project submission. For that, you can count Assignment Help NZ services. Under this platform, you can earn the support of a professional academic writer who helps you to finish your paper properly.

assignment writing help said...

One of the main reasons why students do not want to write an assignment on their own is the lack of interest in that particular subject, language barrier and lacking the ability to present their knowledge in a well formatted content driven paper. Keeping pricing low there are online services like assignment writing help which render support to students in writing their assignments or homework. Panel of writers from across the world and almost every subject on the menu they can help you with every possible assignment you are meant to write.
https://www.allassignmenthelp.com/assignment-writing-service.html

Commercial Pilot Training Program in Canada said...

It is very fun for school students to know everything about Pilot Training and it. When they are introduced to training, they become curious to know more about how it manipulates data and information. As we all know that Pilot Training is an electronic device that stores, retrieves and processes information. Kids are very happy when it comes to practicals, Commercial Pilot Training Program in Canada but as soon as they are asked for pilot training, they panic because training means giving him a lot of time and a lot of research.

Animation Companies said...

CITRUS INK STUDIOS is a one-stop shop for Animation Studios + Visual Effects, Motion Graphics, and VFX Company with the global presence in India.

Commercial Pilot Training Program in Canada said...

One of the main reasons why students do not want a training themselves is lack of interest in that particular subject, language barrier and lack of ability to present their knowledge in a well formatted content driven paper. Commercial Pilot Training Program in Canada. Keeping the pricing low, there are online services like My Pilot Training Assistance that help students with their training. Pilot Training Panel from around the world and almost every topic on the menu They can help you with every possible training.

dgca ground classes in delhi said...

Experts have knowledge from every field whether it is Pilot Training or DGCA, they are post graduate and have great experience in this field. It is a platform on which experts help you with your dgca ground classes in delhi with accuracy. With the help of this platform, you can easily do your training which will help you to get good grades in your academics. Students who get good grades may find it useful.
http://ceaaviation.org/ground-class.php

best dgca ground classes in delhi said...

Experts have knowledge from every field whether it is Pilot Training or DGCA, they are post graduate and have great experience in this field. It is a platform on which experts help you with your best dgca ground classes in delhi with accuracy. With the help of this platform, you can easily do your training which will help you to get good grades in your academics. Students who get good grades may find it useful.

Assignment Help said...

I am really feel glad to read out published post on this blog. Apart from this, I am still waiting other post for escalating my knowledge power. In the time scarcity situation, you can leverage from Assignment Help USA to secure highest grade.

sign and lighting services said...

Pole lights are the most effective way to lighting any size parking lot. You can hire a lighting designer or a lighting firm to help you with this. They are experts in this sector.
parking lot pole, gas station price signs

Costa Rica vacation rentals beachfront said...

Renting a villa on the beach rather than a hotel on the beach will give you far more private access to the area immediately in front of your property. Hotels can be full of hundreds of guests all looking to get a spot on the beach, whereas you can stake your place early in the morning before any other holidaymakers arrive.
villas on the beach in costa rica, house rental costa rica beach