Fon Johnson served as the first athletic director and the first basketball coach for the College for Men in 1955. He oversaw the creation of the football field which was completed in 1955. Gil Kuhn formed the USD Booster Club to help support the football program and coached the inaugural year of football. Desire for a football program was lackluster, however, and the program was discontinued in 1961. USD continued to compete in intercollegiate basketball, baseball, tennis, and golf. The athletic program reintroducing football in 1973 with an anonymous donation of $115,000 also adding soccer around that time. With the advent of Title IX in 1972, the athletics program grew to include female sports. Wendy Gilchrist competed in Tennis in 1971, and Terry Fowler competed as the first female swimmer in 1973. In 1978, the athletics program began moving from NCAA Division III to NCAA Division I, with the exception of the football program which remained Division III until 1993.

USD athletes were initially known as the Pioneers, but in 1961 the mascot changed to the Toreros. Torero is the Spanish word for bullfighter, and was chosen in part as a signifier of courage, honor, and fidelity.

Shortly after the College for Men and College for Women merged, emphasis was placed on the Recreation component of the Athletics Department. The aim was to provide extra-curricular activities for students as well as recreational activities for faculty, staff, and alumni.