How can a service written outside Python or C++ call Triton?
ML System DesignLevel: IntermediateInference client protocols
How can a service written outside Python or C++ call Triton?
Hints
- Consider the wire protocol rather than language bindings.
- Which parts of an inference request must match the model?
Answer
Use the supported HTTP protocol or generate a gRPC client from the service definitions for the target language. Official client libraries are conveniences rather than a requirement to use the server. Check tensor names, shapes, datatypes, timeout and error handling; account for serialization and transport overhead when benchmarking.
Source and licence: NVIDIA — Triton Inference Server FAQ · BSD-3-Clause · answer: Quantoros editorial