High-tenure workers who experience a job loss suffer from large and persistent earning losses. The aim of this paper is to understand and quantify the forces behind these empirical findings. We propose a structural model of the labor market with on the job search, in which firms are heterogeneous in productivity and workers accumulate both general and specific skills while employed. Jobs are destroyed at an endogenous rate due to idiosyncratic productivity shocks and workers’ skills depreciate during unemployment. The model is estimated based via the simulated method of moments using matched employer-employee data from Germany. By matching moments related to workers’ mobility and wage dynamics, the model is able to reproduce the size and persistence of the earning and wage losses found in the data. The results point to the loss of a job at a high-productivity firm as the key force behind the cost of job loss.