(Note: The SAE standards really consist of just five levels. For some reason they've got a "Level 0 - The human driver does all the driving," which is like saying I've got Level 0 superpowers.)
Level 1: "The vehicle can sometimes assist the human driver with either steering or braking/accelerating, but not both simultaneously."
I.e. cruise control or lane keeping.
Level 2: "The vehicle can [control] steering and braking/accelerating simultaneously under some circumstances. The human driver must continue to pay full attention…at all times and [be ready to take over]."
I.e. adaptive cruise control, automated lane keeping, automated emergency braking.
Level 3: "The vehicle can…perform all aspects of the driving task under some circumstances. In those circumstances, the human driver must be ready to take back control at any time when the [vehicle] requests the human driver to do so."
This huge leap between Levels 2 and 3 is where things start to get murky, and where I feel designers need to put in some serious work on the interfaces and communication standards. Imagine you and a friend are up on a scaffold, attempting to install something heavy overhead, where one person has to hold the thing while the other makes microadjustments and drives fasteners--and you're not clearly communicating with each other. Instead one of you just yells at the other when the thing starts to fall. How's that going to go?
Level 4: "The vehicle can…perform all driving tasks and monitor the driving environment…in certain circumstances. The human need not pay attention in those circumstances."
Still murky, this "in certain circumstances" vagueness, and the means for how the hand-off is meant to be executed, and how quickly. Humans strapped inside a two-ton missile that's traveling at 70 miles an hour can either pay attention, or not pay attention. The transition between the two states doesn't come instantaneously, and there are dead people to prove this.
Level 5: "The vehicle can do all the driving in all circumstances. The human occupants are just passengers and need never be involved in driving."
Probably the safest--and of course the most difficult to achieve. Another thing not taken into account here are the darker parts of human nature and what happens when Level 5 cars share the road with non-autonomous cars: What happens when some jerk in a '92 Fiero can identify Level 5 cars and cut them off at will, knowing that the software of those cars has to yield to his aggressive driving?
In addition to a failure of some of the levels to consider human psychology, there are legislative issues: The SAE standards are vaguely agreed upon internationally, yet in the U.S., federal guidance has shifted to state-by-state regulations.
Right now half of the country thinks our President is a lunatic, the other half believes he's a genius. What are the odds we'll get all fifty states to agree on what constitutes a safe autonomous vehicle?