Armour was born in Stockbridge, New York to Danforth Armour and Juliana Ann Brooks. He grew up on his family's farm. Armour was descended from colonial settlers of Scottish and English origin, with his surname originating in Scotland.
Among his first jobs was that of Driver on upstate New York's Chenango Canal which ran through Madison County at that time. At the age of 19, Armour left New York with about 30 other people for California, joining the great California gold rush. Before the journey, Armour “had received several hundred dollars from his parents,” making him, for the most part, “the financier of the party,” according to biographer Edward N. Wentworth. In California, Armour eventually started his own business, employing out-of-work miners to construct sluices, which controlled the waters that flowed through the mined rivers. In only a few years, Armour had turned his business into a profitable enterprise, earning himself about $8,000 by the time he had turned 24.
