We study the singularities of secant varieties of smooth projective varieties using methods from birational geometry when the embedding line bundle is sufficiently positive. We give a necessary and sufficient condition for these to have $p$-Du Bois singularities. In addition, we show that the singularities of these varieties are never higher rational, by giving a classification of the cases when they are pre-1-rational. From these results, we deduce several consequences, including a Kodaira-Akizuki-Nakano type vanishing result for the reflexive differential forms of the secant varieties.