Skip to content
BranchesTags

Repository files navigation

eIQ AAF Connector - REST-based server for ARA-240 LLM inference

License Platforms Language AI/ML BSP

Overview

The eIQ AAF Connector is a REST-based server that enables LLM inference on NXP i.MX processors with the ARA-240 DNPU. It provides a simple Chat Completions-based HTTP interface for serving models to client applications.

graph LR
    subgraph FRDM_iMX["**FRDM i.MX Platform (Host)**"]
        subgraph AAF_Server["**eIQ AAF Connector**"]
                API["REST API Server"]
                Optimum["Optimum Ara"]
            end
        subgraph proxy["**Ara240 Runtime SDK**"]
            LB["Proxy"]
        end
    end

    subgraph Accelerator["**Ara240 DNPU**"]
            Model["Loaded LLM<br/>(model.dvm)"]
    end
        API -->|Prompt| Optimum
    Optimum <-->|socket| proxy

    %% Hardware Communication
    proxy <-->|PCIe | Model

    %% Response Flow
    Optimum -.->|Streaming Tokens| API

    linkStyle default stroke: #FF7400, stroke-width: 2px;

    style FRDM_iMX fill:#EBE7DD,stroke:#262626,color:#262626
    style Accelerator fill:#EBE7DD,stroke:#262626,color:#262626
    style AAF_Server fill:#HEX #262626,stroke:#000000,color:#F7F5F1
    style proxy fill:#HEX #262626,stroke:#000000,color:#F7F5F1

    style API fill:#69CA00,stroke:#262626,color:#262626
    style Optimum fill:#0EAFE0,stroke:#262626,color:#262626
    style Model fill:#FFD800,stroke:#262626,color:#262626
    style LB fill:#F7F5F1,stroke:#262626,color:#262626
Loading

Building the Package

For instructions on building the Debian package from source, see BUILD.md.

Prerequisites

  • i.MX 8MP FRDM or i.MX 95 FRDM board
  • LF6.18.2-1.0.0 BSP (Q1 2026)

The ARA Runtime SDK must be installed on the board.

Note: If using the project from sources, it is highly recommended to use ssh instead of the serial port when running uv commands. If you need to use the serial port, run uv sync commands with --no-progress.

Instructions

I. Setup

  1. Install the deb package
  2. Activate the virtual environment
source /usr/share/eiq/aaf-connector/venv/bin/activate
  1. Run the connector:
connector
  1. By default, the connector will start on 127.0.0.1:8000. To be able to run requests from another device, you can use the --host flag:
connector --host 0.0.0.0

II. Configuration

The server uses a JSON configuration file, server_config.json, to define running parameters and models to load. The description of the main configuration parameters is as follows: The default path of the file is ./config/server_config.json.

Parameter Type Required Default Description Example
log_level string Yes "INFO" Controls server log verbosity "DEBUG", "INFO", "WARNING", "ERROR"
model_config_path string Yes - Template path for model config files (use {} for model name) /usr/share/llm/{}/
model_tokenizer_path string Yes - Template path for model tokenizer files (use {} for model name) "/usr/share/llm/{}/tokenizer"
use_tool_guided_generation boolean No false Enable guided generation for tool calling true, false
llm_params dict Yes - Generation parameters for LLM inference See LLM Parameters below
available_models array Yes - List of available language models See Model Parameters below
semantic_cache array No - Controls semantic prompt caching See Semantic Cache Parameters below

Notes:

  1. To capture requests and responses to the /v1/chat/completions endpoint, use DEBUG as the log level. Then, when sending a completion request, the server will log the latest full request and response JSONs to the latest_response.json file in the root of the project. This can be used for analysis and to easily reproduce issues.
  2. Assuming the models are all kept in storage using the same folder structure, the model_config_path and model_tokenizer_path parameters can be used to provide the default template, allowing users to skip the values in the JSON config for each model.The model name will be used to fill in the {} placeholder in the paths.
  3. Multiple models can be enabled and loaded at the same time. The client must specify which model to use in the request. Make sure the ARA2 model has enough memory to load all enabled models.

The LLM generation parameters are as follows:

Parameter Type Required Default Description Valid Values Example
generate_max_tokens integer Yes 512 Maximum number of tokens to generate in the response Any integer 512
seed integer Yes 42 Random seed for reproducible generation Any integer 42
temperature float Yes 0.0 Controls randomness in token selection. Lower values make output more deterministic 0.0 - 2.0 0.0
top_k integer Yes 0 Limits sampling to top K most likely tokens. 0 disables top-k filtering ≥ 0 0
top_p float Yes 0.0 Nucleus sampling threshold. 0.0 disables top-p filtering 0.0 - 1.0 0.0
repeat_penalty float Yes 1.12 Penalty applied to repeated tokens to reduce repetition Any float 1.12
repeat_last_n integer Yes 32 Number of previous tokens to consider for repeat penalty Any integer 32
ngram_penalty integer Yes 10 Penalty for repeated n-grams to prevent repetitive patterns Any integer 10
frequency_penalty float Yes 1.12 Penalty applied based on token frequency in the generated text Any float 1.12
suppress_penalty float Yes 100.0 Penalty value used to suppress specific tokens during generation Any float 100.0

The model configuration parameters are as follows:

Parameter Type Required Default Description Valid Values Example
name string Yes - Unique model identifier Alphanumeric, dots, hyphens, underscores "qwen2_5-7b"
description string Yes - Human-readable model description Any non-empty string "Qwen2.5 7B instance"
llm_path string No "" path to the llm config file. If missing, the template in model_config_path will be used. "valid_path", "" "/usr/share/llm/qwen2_5-7b/config.json"
tokenizer_path string No "" path to the tokenizer files. If missing, the template in model_tokenizer_path will be used. "valid path", "" "/usr/share/llm/qwen2_5-7b/tokenizer"
type string No "text" Model functionality type "text", "qwen_vl_image", "qwen_vl_video" "text"
tool_calling string No "no" Tool calling support level "native", "non-native", "no" "native"
max_prompt_size integer No 2047 Maximum input prompt size in tokens 1 - 1,000,000 2047
enabled boolean No false Whether model will be loaded on startup true, false true

The semantic cache configuration parameters are as follows:

Parameter Type Required Default Description Valid Values Example
enabled boolean No false Enable semantic caching true, false false
similarity_threshold float No 0.85 Similarity threshold for cache hits 0.0 - 1.0 (inclusive) 0.85
embedding_model string No "sentence-transformers/all-MiniLM-L6-v2" Model to use for generating embeddings Valid HuggingFace model path "sentence-transformers/all-MiniLM-L6-v2"
max_cache_size integer No 1000 Maximum number of cached entries ≥ 1 1000
max_size_percentage_trigger float No 0.95 Percentage of max_size that triggers eviction 0.0 - 1.0 (inclusive) 0.95
eviction_percentage float No 0.1 Percentage of max_size to evict when max_size_percentage_trigger is reached 0.0 - 1.0 (inclusive) 0.1
similarity_metric string No "cosine" Similarity metric for comparing embeddings "cosine", "euclidean", "dot_product" "cosine"

Notes:

  1. The possible values for tool_calling have the following significance:
    • none: No tool calling capabilities. Tools provided in requests will be ignored and no tool calls will be made.
    • non-native: Basic prompt-based tool calling support. Useful for models which do not support native-tool calling.
    • native: Use the native tool calling capabilities of the model by leveraging the model's specific tool calling mechanism and chat template. The model must support native tool calling, otherwise results are undefined.
  2. max_prompt_size limits the input prompt length and context window. It is model specific, so we don't recommend changing it un less you know the exact capabilities of the model. The default value is preconfigured for the Qwen2.5-7B model.
  3. Ensure enabled is set to true only for the model you want to use. Only one model can be active at a time.
  4. If no model has been set to enabled, the server will start, but won't be able to process completions as there are no active models.

Example server_config.json

{
    "log_level": "INFO",
    "model_config_path": "/usr/share/llm/{}/",
    "model_tokenizer_path": "/usr/share/llm/{}/tokenizer",

    "llm_params": {
        "generate_max_tokens": 512,
        "seed": 42,
        "temperature": 0.0,
        "top_k": 0,
        "top_p": 0.0,
        "repeat_penalty": 1.12,
        "repeat_last_n": 32,
        "ngram_penalty": 10,
        "target_token_post_mcp": 1,
        "target_token_pre_mcp": 1,
        "target_prompt_post_mcp": 1,
        "target_prompt_pre_mcp": 1,
        "is_model_specd": 0,
        "frequency_penalty": 1.12,
        "suppress_penalty": 100.0
    },

    "available_models": [
        {
            "name": "qwen2_5-7b",
            "description": "Qwen2.5 7B instance",
            "type": "text",
            "tool_calling": "native",
            "max_prompt_size": 2047,
            "enabled": true
        }
    ]
}

III. Sending requests

  1. After server init, run the following command to check that the server is working:

Qwen2.5 7B template

curl -H 'Content-Type: application/json' -d '{ "model":"Qwen2.5-7B-Instruct","messages":[{"role":"user", "content":"Who are you?"}]}' -X POST 0.0.0.0:8000/v1/chat/completions

Qwen2.5 3B Image template

curl -H 'Content-Type: application/json' -d '{"model":"qwen2.5-image-3B","messages":[{"role":"user","content":[{"type":"text","text":"Describe the image"},{"type":"image_url","image_url":{"url":"/root/test_image.jpg"}}]}]}' -X POST 0.0.0.0:8621/v1/chat/completions
  1. You can navigate to the UI documentation at http://0.0.0.0:8000/docs to interact with the API endpoints and test different requests using an OpenAPI interface.

IV. Guided Generation Notes

  • For models with use_tool_guided_generation enabled, the server will use constrained decoding to ensure tool calls follow the expected JSON schema, provided by the "tools" object in the request. This improves reliability when the model supports native tool calling.
  • Guided Generation will only become active after the model has generated its special tool calling token. (i.e. <tool_call> for Qwen 2.5 models) Until then, generation proceeds normally without constraints.
  • If the model has issues generating the tool calling token, one can try using Prompt Engineering techniques to encourage the model to start its tool calls with the expected token.
  • Converting the tool schema to the State Machine used in Guided Generation is a computationally intensive process which may take a significant amount of time, depending on the number of tools and their complexity. After the schema is converted, it is cached for subsequent requests, which results in much reduced overhead.
  • To help decide if Guided Generation overhead is appropiate for a particular use case, the Connector provides a dedicated benchmark endpoint in POST /bench/guided_gen_perf. Its syntax is identical to the chat/completions endpoint, but it will only run the Guided Generation process without a model (by choosing random tokens), and will return timing information for schema conversion and per-token constraint generation across multiple runs. This endpoint does not use caching, so it will always measure the full overhead.

Feature Matrix

Feature Text LLM Qwen2.5-VL
Tool Calling* ✅ ❌
Token Streaming ✅ ✅
Auto Prompt Truncation** ✅ ❌
Structured Output ✅ ❌
Guided Generation for Tool Calling ✅ ❌
Prompt Caching* ✅ ❌
Video Input ❌ ✅
Image Input ❌ ✅
Conversation Context ✅ ✅
System Prompt ✅ ✅
Semantic Prompt Caching ✅ ❌

*Note: Feature availability may depend on model configuration and capabilities.

**Auto Prompt Truncation automatically drops old messages when conversation exceeds max_prompt_size.

About

REST-based server for serving LLMs running on ARA devices.

Resources

Readme
License
Activity
Custom properties

Stars

3 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages