I'm confused. Quantum computing uses linear algebra and the math of quantum mechanics, to be sure, but that math can be plenty theoretical as well, depending upon your emphasis.
I also have no idea what you're getting at in terms of "knowledge of computers" that bothers you. Does the word "qubit" bother you? Fortunately "qubytes" seem to be beyond current experimental capabilities
. Obviously there are a lots of different areas of research interest and ways to approach quantum computing like any other research field. And yes, you'll probably need to be okay using computers, but isn't that true of almost an area of physics research today?
For the most part, though, people who do quantum computing are either computer scientists, physicists or mathematicians. If you're not comfortable in any of those roles, exactly what place do you see yourself occupying?
If you want to get some sense of what's involved in quantum computing from a theoretical side, I'd recommend taking a look at the canonical quantum computing textbook, "Quantum Computation and Quantum Information" by Nielsen and Chuang.