NLP

[astanisa]

# Use the default TextVectorization parameters
text_vectorizer = TextVectorization(max_tokens=None, # how many words in the vocabulary (automatically add <OOV>)
                                    standardize="lower_and_strip_punctuation",
                                    split="whitespace",
                                    ngrams=None, # create groups of n-words?
                                    output_mode="int", # how to map tokens to numbers
                                    output_sequence_length=None, # how long do you want your sequences to be?
                                    pad_to_max_tokens=True
                                    )

I have got:

[/usr/local/lib/python3.7/dist-packages/keras/layers/preprocessing/index_lookup.py](https://localhost:8080/#) in __init__(self, max_tokens, num_oov_indices, mask_token, oov_token, vocabulary_dtype, vocabulary, idf_weights, invert, output_mode, sparse, pad_to_max_tokens, **kwargs)
    178 
    179     if pad_to_max_tokens and max_tokens is None:
--> 180       raise ValueError(f"If pad_to_max_tokens is True, must set `max_tokens`. "
    181                        f"Received: max_tokens={max_tokens}")
    182 

ValueError: If pad_to_max_tokens is True, must set `max_tokens`. Received: max_tokens=None

Also:

After this code:

from tensorflow.python.keras.layers.preprocessing import text_vectorization
# Setup text vectorization variables
max_vocab_length = 10000 # max number of words to have in our vacabulary
max_length = 15 # max length our sequences will be (e.g. how many words from a tweet does a model see?)

text_vectorizer = text_vectorization(max_tokens=max_vocab_length,
                                     output_mode="int",
                                     output_sequence_length=max_length)

I have got:

TypeError                                 Traceback (most recent call last)
[<ipython-input-39-f5a03fc50979>](https://localhost:8080/#) in <module>()
      6 text_vectorizer = text_vectorization(max_tokens=max_vocab_length,
      7                                      output_mode="int",
----> 8                                      output_sequence_length=max_length)

TypeError: 'module' object is not callable

[mrdbourke]

Hi @astanisa,

What have you tried so far?

Did you try changing the max_tokens value like the error "ValueError: If pad_to_max_tokens is True, must set max_tokens. Received: max_tokens=None" suggests?

[astanisa]

I didn't change anything. It was my fault. I mistyped the code.

[astanisa]

I have another problem:

[astanisa]

It's OK now. I forgot to put [0]

embed_weights = model_1.get_layer("embedding_1").get_weights()[0]

[arghanath007]

I am also getting the same error as well.

If pad_to_max_tokens is True, must set max_tokens. Received: max_tokens=None
I checked the docs and they have updated it from the looks of it.

When I set some specific value for max_token like 10000, the code runs without any errors.

Code

text_vectorizer= TextVectorization( max_tokens= 10000, # Counts of words in the vocabulary standardize="lower_and_strip_punctuation", split="whitespace", ngrams=None, # Create groups of words? output_mode="int", # Mapping the tokens to numbers output_sequence_length= None, # The lenght of the sequences we want? pad_to_max_tokens= True, )

[kpola009]

I facing the same problem did anyone got the fix??

[arghanath007]

Yes, do as the error suggests, set a value for max_tokens. Like I did above in the code block.

max_tokens= 10000