Abstract: To address the challenges in simulating large deformation and fracture behavior of coal rock during coal cutting, this paper proposes a numerical analysis method based on the smoothed particle hydrodynamics (SPH) approach and a hybrid failure model. This method integrates the Drucker-Prager yield criterion and the Grady-Kipp damage model to establish a hybrid failure mechanism capable of simultaneously describing shear and tensile failure. A Fortran solver is developed accordingly. By incorporating techniques such as artificial stress, stress smoothing, and viscous boundaries, stress oscillations and boundary reflection noise in the calculations are effectively suppressed, enhancing numerical stability. The results demonstrate that the model accurately reproduces crack propagation paths and failure modes in pre-cracked rock compression tests and single pick penetration experiments, verifying its reliability in simulating dynamic crack propagation under different loading modes. Furthermore, the model is applied to simulate linear and rotational cutting processes of a pick, revealing the mechanical mechanisms of coal deformation, fracture, and the blocky spalling of concretions under different penetration depths. Two dominant spalling failure modes, shear-dominated and tensile-dominated, are identified, along with their variation patterns with depth.