James Castle was born September 25, 1899, in the small mountain town of Garden Valley, Idaho to Francis J. Castle and Mary Nora Scanlon. The fifth of seven children, Castle was born two months premature. His mother was the local midwife and his father was the postmaster and ran the community post office and general store from the living room of the family home.
Castle was profoundly deaf from birth and did not attend school until he was ten years old, when he was enrolled at the Gooding School for the Deaf and Blind in southeastern Idaho. Castle attended and lived at the school for five years, from 1910-1915.
While at the school, Castle was taught the oral method of communication—sign language was not taught in the public schools at the time—but it is likely he learned sign from the other students who secretly taught each other. Evident in his artwork are various references to specific signs. It is unknown to what extent Castle could read, though his artwork does demonstrate a fascination with written language. Any practical method of communication Castle may have learned at school was lost over the years because it was not practiced at home by his family.
With the exception of his tenure at the Gooding School, Castle resided in only three homes in Idaho: in the small town of Star, and on subsistence farms in Garden Valley and Boise. When Castle’s mother died in the 1930s, his sister Agnes inherited the family farm in Boise where Castle continued to live with his sister, brother-in-law and their four children in the small three-bedroom house.