You have to start from scratch. The training data has many examples of almost directly copied blocks of text and code and the model will tend to just repeat itself after several iterations.