fix: adjust max token size for openai ADA-v2 embeddings

[LeoGitGuy]

Related Issues

• fixes number of input tokens being restricted to 2048 even when using the new "text-embedding-ada-002" which can take 8191 tokens.

Proposed Changes:

The following code shows that the max_seq_len is set to 2048 or lower:

class _OpenAIEmbeddingEncoder(_BaseEmbeddingEncoder):
    def __init__(self, retriever: "EmbeddingRetriever"):
        # See https://beta.openai.com/docs/guides/embeddings for more details
        # OpenAI has a max seq length of 2048 tokens and unknown max batch size
        self.max_seq_len = min(2048, retriever.max_seq_len)

To fix this, I added the user specified parameter retriever.max_seq_len to the function self._setup_encoding_models() and change the max_seq_len when the if clause for the new text-embedding is called, so the function definition will look like this:

def _setup_encoding_models(self, model_class: str, model_name: str, max_seq_len: int):
    """
    Setup the encoding models for the retriever.
    """
    # new generation of embedding models (December 2022), we need to specify the full name
    if "text-embedding" in model_name:
        self.query_encoder_model = model_name
        self.doc_encoder_model = model_name
        self.max_seq_len = min(8191, max_seq_len)
    else:
        self.query_encoder_model = f"text-search-{model_class}-query-001"
        self.doc_encoder_model = f"text-search-{model_class}-doc-001"

How did you test it?

manual tests with text-embedding set to text-embedding-ada-002 and number of tokens set to 8191

Notes for the reviewer

n/A

Checklist

• I have read the contributors guidelines and the code of conduct
• I've used the conventional commit convention for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

this is my first pull request so I'm sorry if I did something wrong, tried my best

[julian-risch]

@LeoGitGuy Thank you for opening this PR. The added functionality looks very good. 👍 I just have a small change request about moving two lines of code so that we use

# OpenAI has a max seq length of 2048 tokens and unknown max batch size
self.max_seq_len = min(2048, max_seq_len)

in the _setup_encoding_models method instead of the init method. My comment below describes that in more detail. Once you add this change, I will merge your PR.

[julian-risch]

Let's add self.max_seq_len = min(2048, max_seq_len) after this line of code instead of having it in the init method. If we add it here, we can remove line 394 and 395 (the self.max_seq_len=... assignment and the comment # OpenAI has a max seq length of 2048 tokens and unknown max batch size) from the init.

That makes it easier to read the code and see where the max_seq_len is set based on the inferred generation of embedding models (Dec 2022 or earlier).

[LeoGitGuy]

Thanks a lot for your reply! I added your proposed change and also fixed a small error since for the earlier embedding models the max_seq_len is 2046 instead of 2048 (see https://beta.openai.com/docs/guides/embeddings/what-are-embeddings). I tested it and hope everything is fine now

[julian-risch]

Looks very good to me and I will merge it now. 👍 @LeoGitGuy thank you so much for your contribution.