Class LocalLlmClient

java.lang.Object
com.codename1.ai.LlmClient
com.codename1.ai.LocalLlmClient
public final class LocalLlmClient extends LlmClient

An LlmClient that runs a GGUF model on the device with llama.cpp. Create one with LlmClient.local(String).

Nothing leaves the device and no network is needed. The model file is loaded on the first request, which can take several seconds for a large model, and stays in memory until close().

LocalLlmClient llm = LlmClient.local(FileSystemStorage.getInstance()
        .getAppHomePath() + "qwen2.5-0.5b-instruct-q4_k_m.gguf");
llm.chat(ChatRequest.builder()
        .addMessage(ChatMessage.user("Name three primes"))
        .build()).ready(r -> Log.p(r.getText()));

The prompt is rendered by the chat template the model file carries, so any instruction-tuned GGUF works without configuration. ResponseFormat.JSON_OBJECT is enforced with a grammar, so the output is always valid JSON. Tools and image parts are not supported and fail the request with LlmException.ErrorType.INVALID_REQUEST.

Availability

Android and iOS. The builders link llama.cpp only into applications that call LlmClient.local(String). Elsewhere, the simulator included, every request fails with LlmException.ErrorType.INVALID_REQUEST; use LlmClient.ollama() to develop against the same model on the desktop.

Requests run one at a time on a background thread, in the order they were made. Results and streaming callbacks arrive on the EDT.

  • Method Details

    • getModelPath

      public String getModelPath()
      Returns:
      the model file this client loads

    • setContextSize

      public LocalLlmClient setContextSize(int tokens)
      Sets the context window. Takes effect when the model is next loaded.
      Parameters:
      tokens - context size in tokens, 0 for the model's default
      Returns:
      this client

    • setGpuLayers

      public LocalLlmClient setGpuLayers(int layers)
      Sets how many layers run on the GPU (Metal on iOS). Takes effect when the model is next loaded.
      Parameters:
      layers - layer count, 0 for CPU only, -1 (the default) for all
      Returns:
      this client

    • setThreads

      public LocalLlmClient setThreads(int threads)
      Sets the CPU threads used for generation. Takes effect when the model is next loaded.
      Parameters:
      threads - thread count, 0 (the default) to let the port choose
      Returns:
      this client

    • isSupported

      public static boolean isSupported()
      Whether this platform can run local models at all. false in the simulator and on any target the builder did not link llama.cpp into.
      Returns:
      true when requests can succeed

    • getProvider

      public String getProvider()
      Description copied from class: LlmClient
      One of "openai", "anthropic", "gemini", "ollama", "local". Used by ChatView and tests to vary behaviour by provider.
      Specified by:
      getProvider in class LlmClient
      Returns:
      stable provider family id

    • chat

      public AsyncResource<ChatResponse> chat(ChatRequest req)
      Description copied from class: LlmClient
      Non-streaming chat. Equivalent to chatStream with a no-op listener but optimized -- the provider skips the SSE response and returns a single JSON object.
      Specified by:
      chat in class LlmClient
      Parameters:
      req - immutable chat request
      Returns:
      asynchronous normalized provider response

    • chatStream

      public AsyncResource<ChatResponse> chatStream(ChatRequest req, StreamingListener listener)
      Description copied from class: LlmClient
      Streaming chat. listener fires for every content delta / tool-call fragment on the EDT. The returned AsyncResource completes with the aggregated final response once the stream ends; cancel it to close the underlying socket.
      Specified by:
      chatStream in class LlmClient
      Parameters:
      req - immutable chat request
      listener - incremental callbacks delivered on the EDT
      Returns:
      asynchronous aggregated response

    • embed

      public AsyncResource<EmbeddingResponse> embed(EmbeddingRequest req)
      Description copied from class: LlmClient
      Creates embeddings for one or more strings.
      Specified by:
      embed in class LlmClient
      Parameters:
      req - immutable embedding request
      Returns:
      asynchronous normalized embedding response

    • close

      public void close()
      Unloads the model. Requests already queued still run first; a later request loads the model again. The worker thread is kept, idle, so a request after close() can never reach the native session on a second thread while the first is still using or closing it.