During the Victorian era, when infant death rates, which before then had gone down, ironically went up due to more medical intervention but not as much knowledge of cleanliness/spreading of germs, it was common for people not to name their infants until they were at least several months of age.
The idea was that the infant had a fairly high chance of dying and supposedly, the parents would get less "attached" to the child if it wasn't named. Of course, the parents were "attached" anyway...but people had precious little at their disposal to buffer themselves against the possibility of tragedy, so this was the way things went for a while.
People often just called the baby "Baby" for its first weeks, months or even first full year of life.
When I lived in northern New Jersey, I visited cemeteries once in a while. There is actually something very peaceful about them. Generally, the farther back I went in the cemetery, the older the tombstones got as of course, the first burials would have taken place fairly close to the church, then spread out as more parisioners died over time. I saw a lot of infant tombstones.

That was the only really sad part. Usually, there would be a last name..."Baby Girl Thompson, in God's Arms" or something like that...but I did actually see some first-name-only tombstones.
The tombstones with only the full name, or even just a first name or, as you saw, "Baby," were almost certainly the graves of very poor people, who couldn't afford anything but the sparest of engravings. Engravings used to be charged by the letter. They probably still are, at least in a general way (amount of space taken up in the engraving, etc.). I even saw some stones that were literally just a marker, with nothing on it at all.