Abstract: In this study, the meshless Smoothed Particle Hydrodynamics (SPH) method was employed to simulate the fracture process of coal mass containing concretions under the action of cutting teeth. Based on the principles of particle discretization and continuum mechanics, the SPH method avoids the mesh distortion problem inherent in the finite element method (FEM) and reduces the complexity of parameter calibration in the discrete element method (DEM). It is particularly suitable for simulating strongly nonlinear behaviors such as large deformation and fracture of materials. This paper systematically elaborates on the basic principles of the SPH numerical algorithm and introduces in detail the implementation of the elastoplastic geotechnical constitutive relationship within the SPH framework, including the establishment of governing equations, numerical discretization, and solution process. To accurately characterize the failure mechanisms of coal and rock under shear and tensile loads, a mixed failure model combining the Drucker-Prager yield criterion and the Grady-Kipp damage model was constructed. Aiming at the potential conflict of failure criteria between the two models in the tensile region, a modification method for the tensile part of the Drucker-Prager criterion was proposed. In addition, artificial stress, artificial viscosity, and stress smoothing techniques were introduced to effectively suppress stress oscillations at interfaces and boundary reflection noise, significantly improving the stability of calculations. On this basis, an SPH calculation program for rock fracture and large deformation was developed using Fortran. This model can naturally simulate the fracture process and spontaneously capture the initiation, propagation, coalescence of cracks, and block spallation without the need for any additional fracture treatment methods. Through numerical simulations of the linear cutting and rotational cutting processes of a single cutting tooth, the mechanisms of deformation, fracture of coal mass, and concretion block spallation during coal wall penetration were revealed. Two dominant concretion spallation modes (shear-dominated and tension-dominated) and their variation laws with cutting depth were identified. Verification through comparisons with pre-cracked rock sample compression tests and single cutting tooth penetration tests demonstrated that the model can accurately reproduce the crack propagation path and failure mode of rock fracture, with simulation results in good agreement with experimental data. This study innovatively combines the mixed failure model with the SPH method, providing an efficient and reliable numerical analysis tool for researching rock-breaking mechanisms under complex geological conditions../t/n.Keywords: Smoothed Particle Hydrodynamics; Mixed Failure Model; Pick Cutting; Grady-Kipp Model; Block Spallation.