Could the universe be a constant size, with everything in it shrinking uniformly?

Scientists tend to state that the universe is expanding. Doesn't it make more sense, uming that nothing exists outside the universe: that it is infinitely small from that viewpoint. It seems more logical to me that everything contained in it is imploding in on itself at the same rate that scientists claim the universe is expanding? If everything within the universe is shrinking, then everything would appear (via observations) to be moving away from each other. If all the elementary particles are shrinking at the same rate the observations would be exactly the same wouldn't they? My reasoning is that if the universe is infinitely small (or large - take your pick) it cannot change size because infinity is constant. Therefore the only way for the observations we make (the expansion) to make sense is for the contents of the universe to be shrinking relative to the size of the entire universe.