TorchServe Files

Highlights Include

• GenAI updates
  • No code LLM deployments with TorchServe + vLLM & TensorRT-LLM using ts.llm_launcher script
  • OpenAI API support for TorchServe + vLLM
  • Integration of TensorRT-LLM engine
  • Stateful Inference on AWS Sagemaker (see blog)
• Support for linux-aarch64
  • CI & nightly regression added
  • Publish docker & KServe images
• PyTorch updates
  • Support for PyTorch 2.4
  • Deprecation of TorchText

PyTorch Updates

• upgrade to PyTorch 2.4 & deprecation of TorchText by @agunapal in https://github.com/pytorch/serve/pull/3289
• Resnet152 batch inference torch.compile example by @andrius-meta in https://github.com/pytorch/serve/pull/3259
• squeezenet torch.compile example by @wdvr in https://github.com/pytorch/serve/pull/3277

GenAI

• Implement stateful inference session timeout by @namannandan in https://github.com/pytorch/serve/pull/3263
• Use Case: Enhancing LLM Serving with Torch Compiled RAG on AWS Graviton by @agunapal in https://github.com/pytorch/serve/pull/3276
• Feature add openai api for vllm integration by @mreso in https://github.com/pytorch/serve/pull/3287
• Set vllm multiproc method to spawn by @mreso in https://github.com/pytorch/serve/pull/3310
• TRT LLM Integration with LORA by @agunapal in https://github.com/pytorch/serve/pull/3305
• Bump vllm from 0.5.0 to 0.5.5 in /examples/large_models/vllm by @dependabot in https://github.com/pytorch/serve/pull/3321
• Use startup time in async worker thread instead of worker timeout by @mreso in https://github.com/pytorch/serve/pull/3315
• Rename vllm dockerfile by @mreso in https://github.com/pytorch/serve/pull/3330

Support for linux-aarch64

• Adding Graviton Regression test CI by @udaij12 in https://github.com/pytorch/serve/pull/3273
• adding graviton docker image release by @udaij12 in https://github.com/pytorch/serve/pull/3313
• Fixing kserve nightly for arm64 by @udaij12 in https://github.com/pytorch/serve/pull/3319
• Docker aarch by @udaij12 in https://github.com/pytorch/serve/pull/3323

Documentation

• Security doc update by @udaij12 in https://github.com/pytorch/serve/pull/3256
• Remove compile note for hpu by @RafLit in https://github.com/pytorch/serve/pull/3271
• doc update of the rag usecase blog by @agunapal in https://github.com/pytorch/serve/pull/3280
• Add some hints for java devs by @mreso in https://github.com/pytorch/serve/pull/3282
• add TorchServe with Intel® Extension for PyTorch* guidance by @jingxu10 in https://github.com/pytorch/serve/pull/3285
• Update quickstart llm docker in serve/readme; added ts.llm_launcher example by @mreso in https://github.com/pytorch/serve/pull/3300
• typo fixes in HF Transformers example by @EFord36 in https://github.com/pytorch/serve/pull/3307
• docs: update WaveGlow links by @emmanuel-ferdman in https://github.com/pytorch/serve/pull/3317
• Fix typo: "a asynchronous" -> "an asynchronous" by @tadayosi in https://github.com/pytorch/serve/pull/3314
• Fix typo: vesion -> version, succsesfully -> successfully by @tadayosi in https://github.com/pytorch/serve/pull/3322

Improvements and Bug Fixing

• Bump torchserve from 0.10.0 to 0.11.0 in /examples/large_models/ipex_llm_int8 by @dependabot in https://github.com/pytorch/serve/pull/3257
• add JDK17 compatible groovy dependency for frontend log4j ScriptFilter by @lanxih in https://github.com/pytorch/serve/pull/3235
• Leave response and sendError when request is canceled by @slashvar in https://github.com/pytorch/serve/pull/3267
• add kserve gpu tests by @rohithkrn in https://github.com/pytorch/serve/pull/3283
• Configurable startup time by @Isalia20 in https://github.com/pytorch/serve/pull/3262
• Add REPO_URL in Dockerfile to allow docker builds from contributor repos by @mreso in https://github.com/pytorch/serve/pull/3291
• Fix docker repo url in github action workflow by @mreso in https://github.com/pytorch/serve/pull/3293
• Fix docker ci repo_url by @mreso in https://github.com/pytorch/serve/pull/3294
• Fix/docker repo url3 by @mreso in https://github.com/pytorch/serve/pull/3297
• Remove debug step in docker ci by @mreso in https://github.com/pytorch/serve/pull/3298
• Fix wild card in extra files by @mreso in https://github.com/pytorch/serve/pull/3304
• Example to demonstrate building a custom endpoint plugin by @namannandan in https://github.com/pytorch/serve/pull/3306
• Benchmark fix by @udaij12 in https://github.com/pytorch/serve/pull/3316
• Update TS version to 0.12.0 by @agunapal in https://github.com/pytorch/serve/pull/3318
• Clear up neuron cache by @chen3933 in https://github.com/pytorch/serve/pull/3326
• Fix Dockerfile fore renamed forks by @mreso in https://github.com/pytorch/serve/pull/3327
• Load all models including targz by @m10an in https://github.com/pytorch/serve/pull/3329
• fix for snapshot variables missing/null by @udaij12 in https://github.com/pytorch/serve/pull/3328

New Contributors

• @andrius-meta made their first contribution in https://github.com/pytorch/serve/pull/3259
• @slashvar made their first contribution in https://github.com/pytorch/serve/pull/3267
• @RafLit made their first contribution in https://github.com/pytorch/serve/pull/3271
• @wdvr made their first contribution in https://github.com/pytorch/serve/pull/3277
• @Isalia20 made their first contribution in https://github.com/pytorch/serve/pull/3262
• @jingxu10 made their first contribution in https://github.com/pytorch/serve/pull/3285
• @EFord36 made their first contribution in https://github.com/pytorch/serve/pull/3307
• @emmanuel-ferdman made their first contribution in https://github.com/pytorch/serve/pull/3317
• @tadayosi made their first contribution in https://github.com/pytorch/serve/pull/3314
• @m10an made their first contribution in https://github.com/pytorch/serve/pull/3329

Platform Support

Ubuntu 20.04 MacOS 10.14+, Windows 10 Pro, Windows Server 2019, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4). TorchServe requires Python >= 3.8 and JDK17.

GPU Support Matrix

Inferentia2 Support Matrix