AWS Machine Learning Blog· Dmitry Soldatkin·· 4 天前AI 评分50
Amazon SageMaker AI 实时端点现已支持 KV cache 感知路由
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
AI 导读
Amazon SageMaker AI 为实时端点推出 KV cache 感知路由(RoutingStrategy 设为 KV_AWARE),依据模型容器上报的缓存状态,把请求发给持有最长缓存匹配且不繁忙的实例。
来源:AWS Machine Learning Blog · aws.amazon.com