Astronomy and astrophysics requires a lot of programming in both observation and theory.
Observation requires reducing data and data mining. Theory often entails numerical simulations. But even if you joined a theory group that didn't do simulations, they will test and visualize data collected by the observationalists to guide their models--which requires programming.
Maybe you'd be more interested in something like instrumentation, which is more hands-on, and requires more use of software to design than, say, writing code.