On the Connectivity of Fiber Graphs