I assume you are referring to Q2 in the 1.4.5 problem set from "Conquering the Physics GRE".
I also found this to be a tricky problem. What I would do is first take the volume integral of the density and set it equal to the mass. Then I solve for A. I believe this is shown pretty well in the solution, giving us
. (Note that the
is because it's a volume integral in spherical coordinates and that the
is from integrating over
and
.
Once you have this you integrate
, where s is not the same as r. This is the really tricky part of this problem. s, as is explained in the solution, is the perpendicular distance from the axis of rotation to a point on the sphere. Moments of inertia are always computed using this, which is also known as a Moment Arm. Thus we can define
Thus, your fully integral would be
.
I hope that this was helpful. If you have any questions please feel free to ask.
Thanks.