Hi Stefan,
Great question! Am I understanding correctly that the only difference between the two model objects is the batch size? If that's the case, you may have some success using Dynamic Tensor Shapes.
Otherwise, if you're comfortable sharing a basic reproduction of the issue in a sample application, we'd be happy to take a look at it.
Hello,
first thing - sorry for taking my time with the reproduction!
I created a very simple project that should show the issue - https://github.com/dudaskostefan/Reproduction138376
The app should show that we need 2 identical model files to handle 2 interactions with the ML Mapping that only differs in batch_size.
Knowledgebase interactions embeds to vector space 300 knowledgebase items (sequences) and user input handles batch_size = 1 because we vectorize user question. If I tried to rely on dynamic tensor shape instead of static tensor shape (in knowledge base preprocessor I changed number of simulated sequences from 300 to 200), I got the error :
It might have been the the current onnx model cannot fallback to dynamic tensor shape but relies on static tensor shape but I tried number of models and still got the same issue, I believe the mapping definition is correct:
Reason, why I cannot update it to batch_size = 301 is because I only need to vectorize knowledgebase once, so I don't want to vectorize 300 KB items all the time. When I tried to only use the model with batch_size = 300 also for user question and PAD the other 299 sequences with nulls, performance was the same as if I was handling 300 items, so this workaround would not work. Also, I was not able to just copy-paste another MLMapping from the first one and just change batch_size because the mapping creates and wires to a new entity (ML_{Input/Output}_Entity_Multibatch_sequence_tokenizer).
Thank you, if you need additional details, I'm happy to provide it!
Stefan