Asking Chat-GPT to re-build in keras-nlp!

[innat]

I asked it and it said
😵‍💫

import tensorflow as tf
import tensorflow_datasets as tfds
import keras_nlp

# Download and preprocess the training data
train_data, validation_data, test_data = keras_nlp.datasets.load_conversation_pairs()

# Create a Tokenizer object with a vocabulary of 10,000 tokens
tokenizer = tfds.features.text.Tokenizer(vocab_size=10000)

# Build the model architecture
inputs = keras.layers.Input(shape=(None,))
encoder_outputs = keras_nlp.layers.Bidirectional(keras_nlp.layers.GRU(64))(inputs)
decoder_outputs = keras_nlp.layers.AttentionDecoder(64, vocab_size)(encoder_outputs, inputs)
outputs = keras.layers.Dense(vocab_size, activation='softmax')(decoder_outputs)

# Compile the model
model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train the model
history = model.fit(
    train_data, 
    epochs=10, 
    validation_data=validation_data
)

# Evaluate the model on the test data
test_loss, test_acc = model.evaluate(test_data)
print(f'Test loss: {test_loss:.3f}, Test accuracy: {test_acc:.3f}')

# Generate text
start_text = "Hello, how are you?"
num_words = 20

# Tokenize the start text
start_seq = tokenizer.encode(start_text)

# Generate the text
predicted_text = start_text
for i in range(num_words):
    input_seq = tf.expand_dims(start_seq, 0)
    logits = model(input_seq, training=False)[0]
    logits = logits[-1, :]
    prob = tf.nn.softmax(logits).numpy()
    index = np.random.choice(range(vocab_size), p=prob)
    predicted_text += " " + tokenizer.decode([index])
    start_seq = tf.concat([start_seq, index], axis=-1)

print(predicted_text)

[jbischof]

That's amazing @innat! Of course, this example is vanilla text generation and does not actually use keras-nlp! (The import for Bidirectional should be from keras proper and AttentionDecoder is an old seq2seq method).

We are actually working on a text generation API with GPT2 to be released in v0.4.1 in Q1 next year. Stay tuned!

[innat]

[Info] https://github.com/hpcaitech/ColossalAI/tree/main/applications/ChatGPT