Probably not that surprising that far more people travel through the Lincoln Tunnel by bus rather than car. So how long before we see restrictions on single-occupant vehicles during peak hours (6-10 AM and 4-7 PM)? Whereby those hours would be restricted to commercial vehicles only - buses, taxis, limos, and trucks.
I can definitely see that happening over the next 5-10 years. And before anybody calls me crazy, allow me to point out what the city of New York did with Times Square and Broadway. If ten years ago anybody had said the city would shut down half of Times Square to traffic and turn it into a pedestrian plaza in addition to long stretches of Broadway, they'd be called crazy. But that's exactly what happened.
The only issue I can think of is the logistics of enforcing such a policy - the XBL is easy because it is only one lane with restricted access guarded by a cop, with a "mercy exit" right there. All of 495 would be a bit more tricky but could probably be done with a few physical adjustments.
In addition, is there any real reason why the XBL cannot be activated during times of heavy inbound traffic on weekends? Not sure if it would make that much of an adverse effect on outbound traffic - certainly can't be any worse than weekday mornings outbound, especially around 7-9 AM.