LSD and marijuana are not the same class of drug, either legally or culturally, and it would greatly improve your ability to make safe judgements safe for you if you became more fluent in the cultural mores of your friend group and the group from which you're selecting potential partners. Personally I've found that preferable. We met online; we could tell by each other's profile that the match looked very, very promising right off the bat. I have friends who drink and smoke and use drugs occasionally, and they're fine people. I even recommend it. Make your decisions based on real knowledge, not abstract, ignorant standards of "influence" that you yourself don't live up to.
Ruling out someone who goes to a weekly wine-and-Italian conversation group and is otherwise a great match for you may or may not be what you want, but it's a risk.
10 Benefits of Dating a Stoner As Told By a Non-Smoker
Do you hate weed because it gives you mad anxiety? I've smoked pot every day for a decade, and most people are outright surprised when they learn that about me. There's plenty of people out there with similar preferences to you, and anyone who isn't an asshole won't have much of a real opinion on it. Does not your relationship with your relationship until it? Some people may view casual sex rather lightly and for others it may be a turn off.