You are correct that the infrastructure is the weak point. (This is true of all sorts of infrastructure, of course -- roads, bridges, electricity transmission, dams, etc.) A proper track signaling system will also include checks for track integrity, but these can be bypassed by sufficiently savvy saboteurs, such as in the 1939 City of San Francisco derailment and the possible copycat 1995 sabotage on the Sunset Limited.