I've always found that it's better to sand between coats. Otherwise, you're only scratching the surface so to speak. Seeing as it's only primer, if it's not too bad, you can get it out during the wet sand. Just speaking from my experience though, I think you're taking the right approach.
Places to go, People to kill, and a Rendezvous with Destiny