Beam Breakup Instability and Coherent Synchrotron Radiation Limits of the Particle Accelerator CBETA