The reason you can't just use the cm is because the moment of inertia is proportional to r^2, not r. If it were linear in r, then you could use cm. Just like how kinetic energy is proportional to velocity squared, you can't just take the average velocity of a system and use that to calculate the total kinetic energy. If you did, you would get zero kinetic energy for a gas, for example. Now, imagine two balls connected with a rod rotating about the center of mass. Using center of mass, you would get zero angular momentum.