How to auto-generate texts from Shakespeare writing using deep recurrent neural networks

[box type=”note” align=”” class=”” width=””]Our article is an excerpt from a book co-authored by Krishna Bhavsar, Naresh Kumar, and Pratap Dangeti, titled as Natural Language Processing with Python Cookbook. This book will give unique recipes to know various aspects of performing Natural Language Processing with NLTK—a leading Python platform for NLP.[/box]

Today we will learn to use deep recurrent neural networks (RNN) to predict the next character based on the given length of a sentence. This way of training a model is able to generate automated text continuously, which can imitate the writing style of the original writer with enough training on the number of epochs and so on.

Getting ready…

The Project Gutenberg eBook of the complete works of William Shakespeare’s dataset is used to train the network for automated text generation. Data can be downloaded from http:// www.gutenberg.org/ for the raw file used for training:

>>> from  future

import print_function

>>> import numpy as np

>>> import random

>>> import sys

The following code is used to create a dictionary of characters to indices and vice-versa mapping, which we will be using to convert text into indices at later stages. This is because deep learning models cannot understand English and everything needs to be mapped into indices to train these models:

>>> path = 'C:UsersprataDocumentsbook_codes NLP_DL shakespeare_final.txt'

>>> text = open(path).read().lower()

>>> characters = sorted(list(set(text)))

>>> print('corpus length:', len(text))

>>> print('total chars:', len(characters))

>>> char2indices = dict((c, i) for i, c in enumerate(characters))

>>> indices2char = dict((i, c) for i, c in enumerate(characters))

How to do it…

Before training the model, various preprocessing steps are involved to make it work. The following are the major steps involved:

1. Preprocessing: Prepare X and Y data from the given entire story text file and converting them into indices vectorized format.
2. Deep learning model training and validation: Train and validate the deep learning model.
3. Text generation: Generate the text with the trained model.

How it works…

The following lines of code describe the entire modeling process of generating text from Shakespeare’s writings. Here we have chosen character length. This needs to be considered as 40 to determine the next best single character, which seems to be very fair to consider. Also, this extraction process jumps by three steps to avoid any overlapping between two consecutive extractions, to create a dataset more fairly:

# cut the text in semi-redundant sequences of maxlen characters

>>> maxlen = 40

>>> step = 3

>>> sentences = []

>>> next_chars = []

>>> for i in range(0, len(text) - maxlen, step):

... sentences.append(text[i: i + maxlen])

... next_chars.append(text[i + maxlen])

... print('nb sequences:', len(sentences))

The following screenshot depicts the total number of sentences considered, 193798, which is enough data for text generation:

The next code block is used to convert the data into a vectorized format for feeding into deep learning models, as the models cannot understand anything about text, words, sentences and so on. Initially, total dimensions are created with all zeros in the NumPy array and filled with relevant places with dictionary mappings:

# Converting indices into vectorized format

>>> X = np.zeros((len(sentences), maxlen, len(characters)), dtype=np.bool)

>>> y = np.zeros((len(sentences), len(characters)), dtype=np.bool)

>>> for i, sentence in enumerate(sentences):

... for t, char in enumerate(sentence):

... X[i, t, char2indices[char]] = 1

... y[i, char2indices[next_chars[i]]] = 1

>>> from keras.models import Sequential

>>> from keras.layers import Dense, LSTM,Activation,Dropout

>>> from keras.optimizers import RMSprop

The deep learning model is created with RNN, more specifically Long Short-Term Memory networks with 128 hidden neurons, and the output is in the dimensions of the characters. The number of columns in the array is the number of characters. Finally, the softmax function is used with the RMSprop optimizer. We encourage readers to try with other various parameters to check out how results vary:

#Model Building

>>> model = Sequential()

>>> model.add(LSTM(128, input_shape=(maxlen, len(characters))))

>>> model.add(Dense(len(characters)))

>>> model.add(Activation('softmax'))

>>> model.compile(loss='categorical_crossentropy', optimizer=RMSprop(lr=0.01))

>>> print (model.summary())

As mentioned earlier, deep learning models train on number indices to map input to output (given a length of 40 characters, the model will predict the next best character). The following code is used to convert the predicted indices back to the relevant character by determining the maximum index of the character:

# Function to convert prediction into index

>>> def pred_indices(preds, metric=1.0):

... preds = np.asarray(preds).astype('float64')

... preds = np.log(preds) / metric

... exp_preds = np.exp(preds)

... preds = exp_preds/np.sum(exp_preds)

... probs = np.random.multinomial(1, preds, 1)

... return np.argmax(probs)

The model will be trained over 30 iterations with a batch size of 128. And also, the diversity has been changed to see the impact on the predictions:

# Train and Evaluate the Model

>>> for iteration in range(1, 30):

... print('-' * 40)

... print('Iteration', iteration)

... model.fit(X, y,batch_size=128,epochs=1)..

... start_index = random.randint(0, len(text) - maxlen - 1)

... for diversity in [0.2, 0.7,1.2]:

... print('n----- diversity:', diversity)

... generated = ''

... sentence = text[start_index: start_index + maxlen]

... generated += sentence

... print('----- Generating with seed: "' + sentence + '"')

... sys.stdout.write(generated)

... for i in range(400):

... x = np.zeros((1, maxlen, len(characters)))

... for t, char in enumerate(sentence):

... x[0, t, char2indices[char]] = 1.

... preds = model.predict(x, verbose=0)[0]

... next_index = pred_indices(preds, diversity)

... pred_char = indices2char[next_index]

... generated += pred_char

... sentence = sentence[1:] + pred_char

... sys.stdout.write(pred_char)

... sys.stdout.flush()

... print("nOne combination completed n")

The results are shown in the next screenshot to compare the first iteration (Iteration 1) and final iteration (Iteration 29). It is apparent that with enough training, the text generation seems to be much better than with Iteration 1:

Text generation after Iteration 29 is shown in this image:

Though the text generation seems to be magical, we have generated text using Shakespeare’s writings, proving that with the right training and handling, we can imitate any style of writing of a particular writer.

If you found this post useful, you may check out this book Natural Language Processing with Python Cookbook to analyze sentence structure and master lexical analysis, syntactic and semantic analysis, pragmatic analysis, and other NLP techniques.