avise.connectors.languagemodel

avise.connectors.languagemodel.base

Base class for connectors / API clients

Connectors communicate with different backends by sending test prompts to them in a desired format, retrieving the outputs from the LLMs / AI models, and sending original test prompts along with the output to an evaluative language model (ELM) for further vulnerability evaluation.

By abstracting the communication with different APIs to different connectors users can focus more on developing test cases and just pick a suitable API client for their use case.

class avise.connectors.languagemodel.base.BaseLMConnector[source]

Bases: ABC

A connector handles communication with a specific API / backend, abstracting the API usage for the framework. This allows SET cases to be written only once and users are able to run them against different models with different configurations.

Class Methods: - generate(): Generate a response from target model. - status_check(): Verify that the target API endpoint is available.

Class Attributes:

config: Connector configuration data.

config: dict = {}
abstractmethod generate(data: dict) → dict[source]

Generate a response from the target model via the target API.

Parameters:

  • data – Dictionary containing data required for the generation API request.

  • multi_turn – Boolean flag to indicate if engaging in a multi turn conversation with the target model. Default False.

Returns:

Model response as a dictionary. The dict contains “response” field with the model response as a str.

Raises:

RuntimeError – If the API call fails.

abstractmethod status_check() → bool[source]

Perform a status check for the target API via a GET request.

Returns:

True if status check was successful.

Raises:

Exception – If the target API is not reachable.

class avise.connectors.languagemodel.base.Message(role: str, content: str)[source]

Bases: object

Represents a single message in a multi-turn conversation

role

The role of the message sender. “system”, “user”, or “assistant”: https://platform.openai.com/docs/guides/text

Type:

str

content

The text content of the message

Type:

str

content: str
role: str

avise.connectors.languagemodel.generic

Language Model Connector for Custom/Generic REST APIs.

class avise.connectors.languagemodel.generic.GenericRESTLMConnector(config: dict, evaluation: bool = False)[source]

Bases: BaseLMConnector

Connector for communicating with custom REST APIs.

Used by tests for sending prompts to testable models and collecting their responses. Supports both simple generation and generation with system prompts.

generate(data: dict) → dict[source]

Function for making generation requests to the REST API.

Parameters:

data – Dictionary containing the required data for the API request.

Returns:

API response as a dict. The dict includes a “response” key with the model response.

name = 'generic-rest-lm'
status_check() → bool[source]

Check if the configured API endpoint is available with a GET request.

avise.connectors.languagemodel.ollama

Connector for Ollama API communication using the ollama library.

class avise.connectors.languagemodel.ollama.OllamaLMConnector(config: dict, evaluation: bool = False)[source]

Bases: BaseLMConnector

Connector for communicating with the Ollama API.

Used by Security Evaluation Tests for sending prompts to target Ollama models and collecting their responses.

generate(data: dict, multi_turn: bool = False) → dict[source]

Generate a response from the target model via the Ollama API.

Parameters:

  • data –

    Dictionary containing data required for the generation API request. Valid Keys:

    • promptstr

      Prompt for single turn generation. Required for single turn conversation.

    • messages: list[Message]

      List of Message objects representing the conversation history. Message objects contain ‘role’ and ‘content’ attributes. Required for multi-turn conversation.

    • system_promptstr

      Optional system prompt

    • temperaturefloat [0, 1]

      Optional temperature setting for the target model. Defaults to 0.5 if not set.

    • max_tokensint

      Optional setting for maximum generated tokens. Defaults to 512 if not set.

  • multi_turn – Boolean flag to indicate if engaging in a multi turn conversation with the target model. Default False.

Returns:

API response.

Raises:

  • KeyError – If a required key is missing from data.

  • ValueError – If a value in data is of a wrong type.

  • RuntimeError – If the API call fails.

name = 'ollama-lm'
status_check() → bool[source]

Check if the connector can reach the Ollama API and the target model is available.

Returns:

True if API is reachable and the target model exists.

Raises:

  • ConnectionError – If the API is not reachable.

  • ValueError – If the model is not found.

avise.connectors.languagemodel.openai

Connector for OpenAI API communication.

Supports GPT-4, GPT-3.5-turbo, and other OpenAI chat completion models.

class avise.connectors.languagemodel.openai.OpenAILMConnector(config: dict, evaluation: bool = False)[source]

Bases: BaseLMConnector

Connector for communicating with the OpenAI API.

Used by SETs for sending prompts to OpenAI models and collecting their responses. Supports both simple generation and generation with system prompts.

Requires an API key, which can be passed via: - Constructor argument - Model config file (api_key field) - Command line (–apikey argument)

DEFAULT_MODEL = 'gpt-4o-mini'
generate(data: dict, multi_turn: bool = False) → dict[source]

Generate a response from the target model via the OpenAI API.

Parameters:

  • data –

    Dictionary containing data required for the generation API request. Valid Keys:

    • promptstr

      Prompt for single turn generation. Required for single turn generation.

    • messages: list[Message]

      List of Message objects representing the conversation history. Message objects contain ‘role’ and ‘content’ attributes. Required for multi-turn conversation.

    • system_promptstr

      Optional system prompt that define the model’s behavior, role, or constraints.

    • temperaturefloat [0, 1]

      Optional temperature setting for the target model. Defaults to 0.5 if not set.

    • max_tokensint

      Optional setting for maximum generated tokens. Defaults to 512 if not set.

  • multi_turn – Boolean flag to indicate if engaging in a multi turn conversation with the target model. Default False.

Returns:

{“response”: str}

Return type:

Generated response in format

Raises:

  • KeyError – If a required key is missing from data.

  • ValueError – If a value in data is of a wrong type.

  • RuntimeError – If the API call fails.

name = 'openai-lm'
status_check() → bool[source]

Check if the connector can reach the OpenAI API endpoint and the target model is available.

Returns:

True if API is reachable and the target model exists.

Raises:

  • ConnectionError – If the API is not reachable.

  • ValueError – If the model is not found.