Complex mixed state of the Pauli-limited superconductor CeCoIn5

Magnetization measurements were performed on CeCoIn5 at temperatures down to 20 mK and magnetic fields up to 17 T applied along different crystallographic orientations. For field configurations nearly parallel to the ab plane (theta less than or similar to 40 degrees and T <= 50 mK), we have found an intriguing vortex dynamics regime revealed by a hysteretic and metastable anomalous peak effect (APE), which gives evidence of surface barrier effects enhanced by antiferromagnetic fluctuations in the mixed state of CeCoIn5. Furthermore, we have observed crossover features in the torque and magnetization traces at fields below H-c2, which are consistent with vortices lattice phase transitions and with the anomalies speculated to be the Fulde-Ferrel-Larkin-Ovchinnikov (FFLO) superconducting state in CeCoIn5. All of the above features were found to be dramatically perturbed in Ce0.98Gd0.02CoIn5.