It is not unusual for a few students from the physics department to "cross-over"
to astronomy/astrophysics research, even when there is a separate astronomy
department. As has been stated several times on this forum, many research
areas are interdisciplinary, and your research advisor may be located in a different
department than the one you were admitted to. If you are uncertain about the
choice between astrophysics and other areas of physics, you should apply to
physics departments to keep your options open. As a physics student, you will
be required to take the core physics courses and pass the physics qualifier, but
you can complete your dissertation in your area of interest. You will probably need
to take classes for a little longer than your astronomy classmates, since they
will usually be taking fewer physics classes and more astronomy classes, but
it will all even out in the end.
One possible negative of this approach is related to funding. Astronomy departments
are usually significantly smaller than Physics departments, and usually attempt to
match admissions to available RA/TA positions much more closely. In addition, they
will give priority to their own students for such positions. Thus, you may find yourself
as a (physics) TA for longer than if you had been accepted directly into the astronomy
Also, I should note that when there is a large disparity in the quality of the physics
and astronomy programs, the astronomy department may be less willing to accept
physics students. So this is not a good approach to try to "backdoor" your way
into a strong astronomy program.