Constant Latency Inference API for String List Processing

anthropic System Design Question Design an inference API with the following requirements: Input is a list of strings, output is a list of strings. The latency of the API should remain constant for list sizes between 1 and 100. You have a group of GPU servers at your disposal. Consider the data flow, database design, and network architecture. Example Input ["hello", "world"]