Don't give in. What others do is irrelevant. Unless son PAID for the car, his insurance, all his expenses, and all his spending money etc... then he has NO rights to the car. There is NO way the car is in ANY way 'his' but he wants you to not think of this or realize this.
Someone under the influence of pot is just as dangerous as someone who is drinking when it comes to driving. Many states treat it as even worse in that you get bigger, harsher consequences. Who is going to pay for his fines and atty fees etc... when he is caught driving under the influence?
Say no, make no happen, and don't mince words. You do not have to give him a reason. It is your car, your belonging, and you are under NO obligation to let him use it at any time. You are also under NO obligation to take him to work, a friend's house or anywhere else. He is an adult. He can save his $$ and buy his own car when he can afford it. He can also pay rent. I realize many people don't think pot is a big deal. Heck, most think alcohol isn't a big deal. But it does become a big deal when you are operating a car or some type of heavy machinery, esp if you have any type of anger issues. Your son won't believe this or thank you, but by not allowing him to drive the car you are protecting him from the lifelong agony of guilt that comes from having hit someone with your car. This is guilt that is a problem for the rest of your life, even with a LOT of therapy, even when the driver was in NO way at fault. I know a man who accidentally hit a teen on a bicycle the first month he had his license. The teen on the bike was dark skinned and wearing all black from head to toe, on a black bike with no reflectors, on a stretch of road with no streetlights. He rode out into traffic to cross the road with less than 20 feet between his bike and the car. There was no way my friend could have NOT hit him. A lengthy police investigation happened because my friend is white and the community screamed racism. The facts were just as clear as day though. There was NOTHING that could have been done by my friend to avoid hitting him. It wasn't much consolation to my friend when he was 16 though. He has had nightmares ever since, and almost 3 decades of therapy. He is a lot better, but he wll ALWAYS feel some guilt and a LOT of regret that it happened at all. He even gave up ice cream for almost 10 yrs because he was on his way to the store just to get some.
If nothing else, you do NOT want that for your son. It took a LOT of hard work for my friend to feel like he deserved anything good in life. He was pretty self destructive for a while. That is NOT what your son needs.
So if you cannot tell him no just because it is what you want to happen, because you know he will NOT refrain from driving while intoxicated/under the influence, will NOT keep people from smoking in the car, will NOT treat your car with respect, then tell him no to do what you can to keep him from having to spend the rest of his life dealing wth that guilt and regret. It was that bad for my friend and he was clean and sober - had never even had beer at a party or stayed at a party if someone started using drugs and he found out. How much worse would the guilt be if he had been impaired? Your son iwll blow this off because he is a kid and he is 10ft tall and bulletproof at this stage. But you know better.
Plus the smell of pot tanks the resale value of the car. If you are driving and the cops pull you over for ANY reason and smell pot in it, even old pot smoke, they can search the car. Even one seed or bit of stem or tiny bit of leaf could get YOU arrested or given a ticket for possession. After all, it IS your car. In some areas if the owner isn't in the car but pot/drugs are found in the car, even in sm amounts, the owner is also given a citation or charged. What would that do to YOUR life and YOUR career? Why would a loving son take that risk? It is unlikely that you would be charged/cited if your son was the driver, but it isn't legal to allow him to use the car if you think he is using. If he is in an accident and the other party can PROVE that you now/knew that he used in the car, and that you knew this before he used the car, then you my bear some of the liability for the accident. I am SURE you have more assets so they would prefer to sue you than your son.
I know it sounds harsh, but it is in your and your son's best interest to not allow him to use your vehicle. He is more than old enough to walk, take a bus, call a friend, ride a bike, or find another way to get to someplace he wants to go to. Let him be an adult and figure his own way to where he wants to be.