…is a wasted opportunity.
Front-end, back-end, it shouldn't matter. You'll come across it again and again.
You don't have to sink a year into it. A couple weekends with a book covering the basics is enough to keep it from being a liability for you. In fact, here's a start.
You should be able to scale and optimize images. You should be able to slice pngs out of a design file. You should be able to navigate layers, add text and images, and save your changes.
Some people hold on to their lack of photoshop experience as a source of pride, as if to say that their so committed to the technical work they're doing that they can't get into photoshop. Seems to me like a rather silly thing to be proud of.