http://jalopnik.com/first-fatal-tesla-autopilot-crash-sparks-nhtsa-investig-1782916450



Taking with a grain of salt the statement about the driver: Since this was a fatal accident, they have no idea what he did or did not notice. They have a system that guarantees his hands were on the wheel, but that doesn't prove one way or the other if he was looking at the road, or if this magic phantom nigh-invisible semitrailer was perceivable by humans or not. Maybe he was staring at his cellphone, or looking at the scenery, or saw the semi and chose not to brake, or whatever.

It's a terrible statement because it tries to imply that their image recognition software is as good as the perception of the human driver. Sorry, no. What they're really admitting is that their image recognition software was so poor that it couldn't tell the difference between the sky and a semi.

Lesson learned: Software image recognition has a long way to go, you can't depend on it yet. You need other systems to scan for objects in your environment.

So, don't cross the street in a white shirt if a Tesla is around?

Agreed, Tony. I worry that this is going to set back autonomous driving, or if it's just what we'd expect to see at this point in its development.

It's undeniable that Tesla owners have been irresponsible with the autonomous driving mode - Youtube can tell you - but you're right, it's hard to know what happened in this case. I guess you'd have to ask: if this exact same scenario had happened in a car that wasn't autonomous would it have happened?

Most people know that cruise control still requires vigilance. I find it scary that Tesla owners don't treat this brand new feature the same way. The company has made it extremely clear that they should.

I really don't want to cast blame on the victim. This is a terrible thing to happen and I feel terrible for him and his family. I'm just trying to figure out how I feel and think about this event. In the long run, I think it's imperative that we develop autonomous vehicles, but to my mind the biggest question is what happens when these accidents occur (and they will). It's so easy to blame a drunk or reckless driver for accidents they cause. But if I'm sitting in a Google car on my way to work in 2025 and the computers inside make a mistake and drive me off the road, how do we respond to that?

Looks like the driver might have been watching a DVD at the time.

Wonder if this crash might be down to the same apparent lack of sensor coverage that allowed one to drive its windshield into the 'overhang' of a parked semi trailer while parking.

The crash may be, but the fatality is down to the fact that US trucks aren't required to have side guards, which would have prevented the car from going under the truck.

Don't worry, I'm sure we won't require them once we are out of the EU.

It would require an awfully strong side guard structure to withstand the highway speed right angle impact of a heavy low slung automobile.

Even if the trailer substructure was able to prevent impact from submarining underneath, the sudden deceleration of the car from highway speed to stopped would be similar to head on collision or hitting fixed roadside structure given the weight difference between car and truck (I do not have any information regarding the actual weight of that particular trailer).

The fatality may have still occurred given the speed involved.

OK, so it might not have prevented the submarining in this case, but...



I checked how strong they're supposed to be. Per 89/297/EEC, the side guards are required to be "capable of withstanding a horizontal static force of 1 kN applied perpendicularly to any part of its external surface...".



Granted, but the crumple zones on the car are probably better than the crumple zones on a human head.



The statement from Tesla states:



Even if this particular crash wouldn't have been mitigated by the side-guards, that's no reason not to fit them.

Edited: Tesla claim it would likely not have been fatal

I think it's a setback for Tesla, but it points out how Google's system relies on things other than image recognition: Laser and radar scanners. I found it interesting to read Matthew Inman's experiences just a couple days after reading about the Tesla accident. He describes how the car's radar system noticed a bicyclist behind a hedge before anyone/anything could have gotten a visual.

Something else I heard... I heard this through word of mouth rather than reading about it, so I can't verify its veracity, though it sounds logical...

Supposedly contributing to the Tesla accident was the need for the image recognition to be able to tell the difference between signs/overpasses/etc and objects in the car's path. Supposedly, because the side of the semi was the same color as the sky, the only part of the semi that was identified as looming closer was the top edge of it. So the software thought that the approaching high-positioned horizontal line was merely a roadsign, a bridge, or powerlines. It decided that it was something that was OK to drive under.

The need for that kind of mitigation in the software is critical, because if it wasn't there, the Tesla would be slamming on its brakes every time it passed under a roadsign or a bridge.

The discussion about trucks needing side guards is interesting, but it's unrelated to this situation, it doesn't address the root problem. The problem is a system that depends on unreliable image recognition software rather than more reliable rangefinding technology.

I LOL'd. Nice callback.

I've not seen anything that guarantees his hands were on the wheel. All the Tesla videos seem to show that definitely isn't the case.

My interpretation.

Tessa's statement on the crash (if you read the entire text of the statement quoted in the article at the top of this thread) describes their system. If the driver's hands aren't on the wheel it reminds the driver and also keeps slowing down. I don't know the system's details but it was described in their statement.

My point was that such a system, however it works, still doesn't guarantee he was looking at the road.

As a huge Tesla fan, I've been reading all sorts of articles on this topic. Personally, I am annoyed as to how the tone of most of them seem to imply, if not say it openly, that this accident is somehow related to Tesla autopilot feature.

It is not.

Let's put it in what seems to me the right perspective:

Tesla Autopilot, as amazing as it is already, could not prevent this accident, caused by humans. So, hopefully, it will soon improve and expand the range of human-caused accidents it can prevent.

Different topic: it is possible this human adopted a particularly silly behavior and decided to give up driving the car and instead rely entirely on an car feature designed to assist the driver in case of emergencies. That is quite irresponsible. If that is the case, than yes, here's one more reason for Tesla to keep improving its Autopilot system to maybe also cope with such human mistakes.

My understanding is that the feature is not intended to assist in emergencies at all. It is to assist with non-emergency driving conditions and requires the driver to take full control the instant the emergency situation begins to occur.

I recently read a well written article (cannot find the link at the moment) which describes similar problems with aircraft auto pilot situations where the pilot must suddenly take control from the automatic system. Apparently there is a roughly 15 second to one minute time period where the pilot is transitioning from not actually flying the plane to fully understanding what is going on and then figuring out what the pilot must now do. During that transition time neither the pilot nor the automatic system is capable of fully responding to the situation.

There appears to be a similar (perhaps shorter) time period when the non-attentive driver in an assisted driving car must suddenly respond to alerts from the automatic system (or perhaps no alerts at all if the automated system is momentarily fooled or confused) and then figure out what the problem is and what to do about it. That driver control transition time period is likely to be multiple seconds, which may be longer than the time available before something really bad happens.

This problem may turn out to be intractable, which may underlie the expectation from some companies that only fully automated driving will be considered 'safe enough'.

If the car is only capable when the driving is easy and requires the human to suddenly take control when it gets difficult, that is precisely the moments when humans often make errors. Adding the automated mode to human control transition time factor seemingly would exacerbate the human error rate.

I think what he meant was that most of the car features related to this are to detect critical situations, warn the driver of them, and react autonomously (eg. emergency braking) if the driver fails to respond.

Then somebody said, "Hey, it can practically drive all by itself", so they added that feature on top of the emergency assists.

Cheers

Yeah, the "auto-pilot" is essentially VERY advanced cruise control and should be treated as such.

I've actually driven a Tesla in autonomous mode. A friend of my mother has one and he took me out in it. I only drove with it on for about half a mile and it was certainly impressive but I would NEVER surrender to it completely and assume that I could take my eyes off the road. The best I could describe it was, again, advanced cruise control. It was really cool, but it wasn't a self-driving car.

Precisely.

Current Autopilot will eventually lead to actually self-driving cars, of course, but even then, the point in time in the future when the vehicle CAN drive itself, that will not imply that the driver becomes a passenger and accidents are considered legally (or morally?) caused "by the car". That would be a further step in the future, when society decides we can rely entirely and only on technology to control a vehicle.

TODAY, Tesla Autopilot is not such thing, neither in theory nor in practice, even though practically it can do a lot of the driving.
I am not sure what you mean, K447, but my understanding is that Tesla Autopilot is exactly an advanced system to assist in emergencies:
- car can follow lanes in most cases, so if you get distracted or fall asleep it will keep the car in the lane for you
- car can break for you, if you get distracted or fall asleep
- car can assist you in changing lanes for you, insuring nobody is coming from behind, adjusting speed and steering, etc.

The fact that these things may equate to "self driving" to most practical purposes, does not mean that IS a self-driving system.

Maybe in 5 years. Maybe in 10.

Today, in a Tesla car you are required to keep you hands on the steering wheel, not watch Harry Potter.

I LOVE where Tesla is going and what they're doing, but I would never watch a movie in a 2016 Model S instead of keeping my eyes on the road.

Which is why this accident was not caused by the Autopilot system, but by human misunderstanding of what the technology is capable of and what its current purpose is, in spite of all disclaimers.

Having said that, I agree that how real automatic piloting systems and human action overlap/interact is non-obvious - your point is very very interesting.
I don't think it applies here though.

Perhaps it does.

Consider that the closer to genuinely trustworthy the automated (as in highly driver assisting but not totally autonomous) driving system becomes, the less response time the human will have when the automated system suddenly discovers it has misunderstood the situation or suddenly abandons its previously considered valid up to that very second understanding, and now throws control (and responsibility) upon the driver, perhaps without any prior warning.

I posit that as the automated driver systems become better they may 'hang on to' vehicle control much deeper/longer into a difficult situation before realizing it actually does not have the capability to exit the current specific situation without high risk of a (machine driving controlled) poor outcome.

If the system is far from perfect and the driver is accustomed to frequently taking over from the machine, each subsequent control handoff might be expected to be similarly routine. If the system is very very good and sudden handoffs back to the driver become uncommon, the likely ability of the driver to become quickly situation aware and respond correctly is lessened.

Perhaps the Tesla system has already become good enough to create this risk. If so, making the driving assistance system better may simply increase the risk of driver failure during unexpectedly poor outcome (to the driving system) events.

People are half distracted by their phones in normal cars already. It's not reasonable to expect them to be anything but 100% distracted in a self driving car...

Yep, all that poxy red tape

I have a friend with a Tesla, he texts and emails whilst driving (on autopilot) once the feature's there I guess you kind of assume that it's infallible.

K447, I understand what you're saying. Here's where I don't agree.

Your point applies to Airplanes, where "Autopilot" is supposed to be used. Similarly, it will apply to cars 10 years from now, when "Autopilot" will be officially a thing, regulations of sorts will be in place, and a precise expectation will be there on how to use that.
In that context, it is interesting to see how all parameters work together. Quality and extensiveness of Autopiloting (sensors, logic, hardware, etc), human interaction (how will we, drivers, be trained to use the Autopilot in driving schools, for example), ways the Autopilot will adopt to "hand over" control and/or alert humans, etc.

Today, it is a whole different story. Tesla nowhere ever tells you you can let the car drive itself. Laws and regulations, when they exist, tell you not to. I would personally add: common sense tells you not to.
Now, the accident here was caused by a human dramatically overestimating the system capability, or, maybe, his own capability to intervene.
If I had to find a parallel to something currently familiar, I'd actually think of drink and driving, or using drugs and driving. When intoxicated, people assume they can drive, they are capable to react and evaluate physics around them, as well as to have common sense and sound judgement. That is clearly a severe misjudgment. Same goes here: this person severely misjudged what the system was for and is capable of.

The problem I see here is entirely human, at the individual level, and possibly at the society level - we don't fully understand what this stuff really is and how we're supposed to use it (and, maybe, lack of regulations may be a contributing factor. Maybe.). There's possibly a lack of expectations on us, when we are on cars with such features: "what am I supposed to do, what is right?"
I also agree a lot of Tesla owners are behaving irresponsibly.

But all of this is human error, not technology issues.

Only if you're a f**king idiot !

I didn't say he was clever!

My brother has just put down a deposit on a new S and he wouldn't even try the autopilot on his test drive, he thinks the whole thing is mental.

I'm hoping to order an X later this year, will definitely go for autopilot but I'd use with caution.

Looking at the Tesla forums it appears that plenty of "features" like the music player are still in a beta state, so their approach to car building is like that of a software vendor.

Perhaps the terminology contributes to confusion. If a car is actually self-driving, but it is not sufficiently good that the driver becomes a passenger, then is it actually self-driving?

My point was that if the car is not entirely self-driving capable, that is, it requires the driver to be always situationally aware and ready to take control at any moment, yet the driving assistance is quite good and continues to improve, then the problem of driver inattention and non-readiness to take control increases as the car ability becomes better and actual control handoff to the human driver become less frequent.

To me it is not at all clear that the current approach to self-driving will inevitably lead to completely self-driving (autonomous) capable. Effectively the Telsa style approach uses the driver as the fail safe 'airbag' for the technology. When the technology runs up against situations where it cannot continue driving with confidence it utilizes the human as backup, an escape route for the tech.

The gap between 99% (or whatever) capable and 100% capable (driver as passenger) may prove to be a wide chasm. Much of this comes from the roads themselves which do not have anywhere near the design, configuration and supporting technology to provide highly reliable information to the car. Another portion comes from the mix of human piloted and varying degrees of automated driving vehicles. And yet another missing piece is technology on all the other cars to provide reliable information on the location, configuration and intentions of those other vehicles.

Some of the methods utilized in Tesla type self-driving cars amounts to 'cheat codes', such as following the tail lights of the car ahead in poor weather. We have all done this in poor visibility driving at speed, and we know that there is some real risk of the driver in front going the wrong way (for us) and following them into a poor vector.

Imagine that the car in the linked video (audio can be muted) was being followed by a 'self-driving' car in poor weather at night. Could the self-driving car simply follow the first car into the water? Would it suddenly hand off control to the human driver half way down the embankment?

https://youtu.be/4NJmB1F2mdE

Construction, temporary road changes, road damage (which can occur suddenly) and non-standard road configurations and other weird problems can conflict with GPS mapped routes and 'normal' visual driving clues.

https://youtu.be/9S26YPzLiDQ

https://youtu.be/u3_5qYBk6Cw

Terminology is misleading, I agree. Having followed very closely Tesla in the last two years, I only focused on what Tesla specifically explains "Autopiloti" is today. I just realized, by reading this thread, that "Autopilot" may in fact suggest that the car drives itself now, in every imaginable way. It will be the case. It is not, yet.


I would say definitely not. In principle, a car is a self-driving car (meaning 100% such) when it drives like or better than humans in all possible conditions. While it may be hard to actually asses that, in principle that seems to me a reasonable and simple definition. It also seems to me that is the meaning Tesla assigns to a fully self-driving car, everywhere they mention the idea.


That is their goal, as Elon Musk stated repeatedly. He claimed they'll succeed in this, from a technological perspective, in 3-5 years, definitely no more than 10. That does not mean that autonomous cars will be legal at that point, though. Statistics will need to support the fact that cars do in fact drive themselves better than humans in all conditions before institutions make it legal for a human to give up control. At that time, though, a number of things would need to change: training drivers to begin with, for example. Codes and regulations, etc.
Until then, humans are responsible what what their cars do.

Interesting related point: technologically, Tesla claims however that a real self-driving car is a solved problem. All they need TODAY is more computing power and improved sensors in some areas. In other words, it is just a matter of when, not of how.
This is FYI, not to say one must believe Tesla, necessarily. I personally do believe they will succeed, but, certainly, it is just my best educated guess.


Honestly, I don't think so. They never said such a thing, or even suggested it - admittedly using the "Autopilot" name for their technology may be misleading. But, Tesla never stated any such thing, they keep repeating the opposite, that is that humans should drive in real life, out of testing environments. You're even required to keep your hands on the steering wheel or the car will progressively slow down to stop.
This is precisely because of some of the limits of the technology you point out.

Please, also notice that more computing power and better sensors will - I think - address your concerns about roads.
As to the mix of human-vehicle intervention, I am not sure I necessarily share your concern. There MAY be issues the moment in the future we start to rely on slef-driving, but it would be interesting to see what interface/interaction designs will be there to address such issues. Keep in mind that we've been living in a world where humans and machine share control for decades now, already, in various degrees. Even a completely mechanical car is a layer between you and the road you do not control to a large extent, and yet we successfully rely on that on a daily basis. I think you're deliberately drawing a line in a world where machine control and human control already fade into each other. No?

My current perspective is that the layers of driver assist and automation that existed prior to the current 'auto-pilot' level could allow the driver to become distracted or reduce attentiveness, but the current and near future self-driving capabilities are exponentially more so.

For me these levels do not fade into each other on a smooth continum. The current capabilities of the Telsa car (not the company's characterization or descriptions of their products, more the ways people are actually using them) is a step change in how people are perceiving this stuff.

Even the most advanced cruise control and lane departure warning equipped car would run off the road in mere minutes or even seconds of sustained driver inattention. Falling asleep was often fatal.

I understand that Telsa, the company, makes great efforts to dissuade people from over-relying on the technology. But Telsa, the actual car, seduces (some, perhaps many) people into trusting the apparently impressive capabilities beyond what is actually 'safe'.

Self-driving cannot be a 'solved problem' if the necessary sensors and compute power are not yet available. It may become evident at some future moment that the current Tesla technology approach is inherently limited and that even stupendous improvements in sensors and compute will not sufficiently bridge the residual risk gap to allow regulatory change and trustworthy self-driven activity, especially in mixed traffic environments with human drivers in other non-advanced vehicles.

When Tesla defines a capability as 'Beta' that to me means they are indeed utilizing the human as a fail-safe wrapper for the technology. The person must be there to take over when the tech fails. The safety and regulatory threshold for removing the Beta asterisk from self-driving appears to be very, very high. It is not a given that more intensively pursing the current path will lead to the desired outcome.

This Critique of NHTSA and SAE "Levels" of self-driving may be of interest.

I note that the fatality that initiated that thread occurred under conditions the author rates as hard and very hard.

My view of Elon is that he is personally very comfortable with high degrees of risk. He is willing to bet the company and all stakeholders that this Tesla experiment can be successful.

I guess we have to agree to disagree, then

I actually do agree with your point that



My point is that that is a human issue, not a technology one, just as texting while driving is a human issue and not a problem inherent to mobile phones.

Which does not mean that phones should not be designed to cope with that problem of course, if at all possible; and, Tesla will be upgraded progressively to cope with human wrong behavior (starting from next to be release software v 9.0, actually!).

I also want to comment on this:


What that means is that Tesla knows how to solve a problem, in software terms. They need a powerful enough computer for such technology to be fast enough to be useful. Consider that we're talking of a very fast response to the environment. This is the core of their cooperation with AMD, actually.

Knowing how to solve a problem and not having a powerful enough computer to perform all calculations is a common situation in science, actually.

Agreed.

Certainly appears to be the case, looking at X owners experiences at the Tesla Owners forum it looks like they're delivering cars that just would never make it out of the factory in a normal manufacturer.

The fact that the car is totally awesome makes up for this for most people, but it is a very different attitude to most major manufacturers.

In think autonomous will ultimately make roads safer, but how long that will take is the question.

I think the problem is that Tesla is treating an automobile as a software product, and letting the public beta test it. They have been proclaiming how their cars have been updated with autopilot, while in fine print saying don't trust it and pay attention while the car is driving.

The auto industry doesn't work like that, at least in America, for what I would think are obvious reasons. No longer is beta software something mostly innocuous, where if it goes wrong it is annoying, but rarely more. Now the safety of not only the driver, but others on the roads are affected.

Putting a feature like autopilot into a car and not expecting it to be abused is naive. It needs to be bulletproof if they are making the claim the car drives itself.

There is a related but separate factor, that of driver competence reduction precisely as a result of frequent exposure to 'automated' self-driving. The driver, who is currently expected to take manual control at the very moments of difficult driving and unexpected situations, is actually less competent to do so as the proportion of driving time under automated control increases.

This 'Glass Cage' article describes the lessons of highly automated aircraft and the correlation effect on competent pilots without recent exposure to manual flying that rapidly degrades their abilities.

Combined with the human control transition time problem, I anticipate more 'self-driving' car negative outcome news events as the cars get better and more people use the auto-pilot modes more intensively.

As an aside, I find it remarkable that Tesla (the brand) is seemingly able to avoid deeply negative press despite documented crash and near crash events, quite different from the 'sudden acceleration' media frenzy that Audi (and other brands) have suffered through in the past. Perhaps the 'press' (such as it is these days) is mostly happy (profitable) using Tesla as a positive news story.

This is interesting, as I largely agree, but from a radically different perspective.

Certainly Teslas are "computers on wheels", as they described their cars. And certainly that is a unique view in the automotive industry. Tesla does releases "beta" products and lets users test them, and does so openly. They actually advertise this quite a lot.

Per se, I think that is good, actually, and also unavoidable considering how the industry is obviously evolving. Even without Tesla pushing in that direction more than anyone else, such change in the industry was/is easily predictable.

That does not mean less safety, though. Equating "beta" to "unsafe" is wrong, I think.
When it comes to safety, I consider Teslas significantly safer than virtually all competitors' cars. Statistics on Autopilot Teslas seem to prove they ARE in fact safer. If then you consider how irresponsible many Tesla owners are being, I am impressed at how good the Autopilot is.

Also, I don't think other less software-minded manufacturers are bringing cars to market that are less "beta" then Teslas. I'd think the opposite, both when it comes to safety and when it comes to other features. My 2003 Audi A3 had a poorly designed clutch that would let 1st gear stuck (manual transmission) and traction would be there even with clutch pedal all down (!!), or, it cut my hand (blood on it) because of poorly metal finishing on the shift lever (I almost ended up in the other lane because of the sudden pain and surprise). And, these are two of the most popular issues with that "traditional" car, if you go and read forums of those years.

I think software for self-driving cars is a prime example of software that can't be developed in the public-beta, agile, MVP, evolution-y kind of way which web development has made so fashionable. A half-arsed Facebook is a lot better than nothing and entices users in, but a half-arsed self-driving car is worse than nothing, as human nature inevitably leads to people trusting it more than it deserves, despite the fine print.

Peter

I wonder how much testing they did of this on real roads before releasing it. I have to imagine it wasn't a fraction of the amount that Google has been doing.

On how safe it actually is. I agree with Vanity Fair.

http://www.vanityfair.com/news/2016/07/h...=social_twitter

I'd agree with that too.

This is another angle on the autopilot story.

Someone on twitter linked that story and said, "so does this bring our net Tesla fatalities back to zero now?"

I saw a fantastic talk about the interplay of ethics with this (and other internet-of-things stuff) at Open Source and Feelings in Seattle a few weeks back. When it's online, I'll link it.