Skip to content

FastNN Client

FastNNClient

class fastnn.client.FastNNClient(url='127.0.0.1:8000', model_name='distilbert-squad', model_version='1', client_type='grpc', verbose=False)

FastNN client class for pushing requests to the Triton Inference Server

Usage:

>>> client = FastNNClient(url="127.0.0.1:8000", model_name="distilbert-squad", model_version="1")
>>> client.request(batch=batch)

Parameters:

  • url - String url of Triton Inference Server. Defaults to 127.0.0.1:8000
  • model_name - String name of model in model_repository directory
  • model_version - String model version name
  • client_type - A string for choosing between http and grpc protocol ["grpc", "http"] (Defaults to "grpc")
  • verbose - Bool for log verbosity
request(self, batch, binary_data=False)

Runs a request with the batch input that can be generated from a FastNN Processor

  • batch - Tuple of torch tensors, typically batch inputs from a dataloader
request_grpc(self, batch, binary_data=False)

Runs a grpc request with the batch input that can be generated from a FastNN Processor

  • batch - Tuple of torch tensors, typically batch inputs from a dataloader
request_http(self, batch, binary_data=False)

Runs an http request with the batch input that can be generated from a FastNN Processor

  • batch - Tuple of torch tensors, typically batch inputs from a dataloader