Milesoft Vertex AI Extension (vertex-ext)

1. Overview & Purpose

The Vertex AI Extension (vertex-ext) is a lightweight, secure, and production-ready JVM wrapper library around the official Google GenAI platform designed specifically for the Milesoft ecosystem. It simplifies integration with Google's state-of-the-art Gemini LLM and vision models hosted on Vertex AI, while providing critical JSON sanitization, HTML block extraction, and stream-handling utility functions.

By encapsulating Google's raw developer libraries and offering a unified service-and-utility layer, the extension ensures architectural consistency and eliminates boilerplate code across downstream applications.

Key Benefits & Core Capabilities:

• Unified Gemini Client Lifecycle: Seamlessly manages the lifecycle of the underlying com.google.genai.Client and ensures safe disposal of connection channels using close-safe JVM patterns.
• Multimodal Visual Reasoning: Offers native support for vision prompts by downloading remote/public image assets, extracting MIME types, and streaming them as binary payloads (inlineData) directly to multimodal Gemini models.
• Robust JSON Reconstruction & Sanitization: Features LlmUtils.toJson which scans text outputs, isolates the outermost brace boundaries, and applies a security-hardened json-sanitizer to automatically repair syntactic flaws (e.g., mismatched quotes, trailing commas) for direct mapping with Jackson.
• Html Content Extraction: Easily isolates HTML structures from markdown code fence segments (```html) to prevent formatting bleed-through on the frontend.
• Standard Configuration Integration: Implements out-of-the-box support for Spring dependency injection, standardizing dual (fast vs. complex) model configurations and qualifier routing.

---

2. Installation & Dependency Configuration

The Vertex AI Extension is published in our secure Google Cloud Artifact Registry repository.

To pull this dependency into your downstream application, ensure you apply the Google Cloud Artifact Registry plugin and configure your maven repositories properly:

2.1 Gradle Setup (build.gradle)

plugins {
    id "com.google.cloud.artifactregistry.gradle-plugin" version "2.2.5"
    id "java"
}

repositories {
    mavenLocal()
    mavenCentral()
    maven {
        url "artifactregistry://us-west2-maven.pkg.dev/milesoft-repo/repo-maven"
    }
}

dependencies {
    // Required base foundation libraries
    implementation "io.milesoft:commons:2.0.1"
    
    // Milesoft Vertex AI Integration Extension
    implementation "io.milesoft:vertex-ext:2.0.1"
}

---

3. Configuring Vertex AI in Spring Boot

To enable proper dependency injection of the LlmService within your Spring Boot application, define it as a managed bean.

3.1 Properties Configuration

In your environment properties files (e.g., application.yml), declare your Vertex AI targets:

vertex:
  project: "your-gcp-project"
  location: "us-central1"
  model: "gemini-2.0-flash-lite"

3.2 Standard Bean Configuration

Register the service class as a Spring @Bean inside your configuration layer:

import io.milesoft.vertex.services.LlmService;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class VertexConfig {

    @Bean
    public LlmService llmService(
        @Value("${vertex.project}") String project,
        @Value("${vertex.location}") String location,
        @Value("${vertex.model}") String model) {

        return new LlmService(project, location, model);
    }
}

3.3 Advanced Dual Model Configuration

For workflows that balance latency with analytical complexity, you can register multiple LlmService instances using Spring's @Qualifier annotations:

vertex:
  project: "your-gcp-project"
  location:
    fast: "us-central1"
    complex: "us-west1"
  model:
    fast: "gemini-2.0-flash-lite"   // Low-latency, cost-effective formatting & search tasks
    complex: "gemini-2.5-pro"       // Deep visual layout analysis & technical reasoning

import io.milesoft.vertex.services.LlmService;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class VertexConfig {

    @Bean
    @Qualifier("complexLlmService")
    public LlmService complexLlmService(
        @Value("${vertex.project}") String project,
        @Value("${vertex.location.complex}") String location,
        @Value("${vertex.model.complex}") String model) {

        return new LlmService(project, location, model);
    }

    @Bean
    @Qualifier("fastLlmService")
    public LlmService fastLlmService(
        @Value("${vertex.project}") String project,
        @Value("${vertex.location.fast}") String location,
        @Value("${vertex.model.fast}") String model) {

        return new LlmService(project, location, model);
    }
}

---

4. API Reference Guide

4.1 LlmService

The central orchestration service used for interacting with the Google Vertex GenAI API. It manages connection clients internally and handles payload structures automatically.

Method Signatures	Return Type	Description
LlmService(String projectId, String location, String modelName)	Constructor	Instantiates the service. All three parameters are mandatory, non-blank strings (validated via preconditions).
generateText(String prompt)	String	Dispatches a simple text prompt. Uses default configuration (candidateCount(1)) and returns the resulting raw text.
generateText(String imageUrl, String prompt)	String	Downloads the image, automatically detects the image type/MIME, creates inline binary data, and dispatches a multimodal vision request.

4.2 LlmUtils

A thread-safe collection of static utility methods for sanitizing, transforming, and correcting raw LLM-generated responses before usage in application code.

Method Signatures	Return Type	Description
toJson(@Nullable String response)	String	Extracts the first bounded JSON block (from { to }) from a response string and applies security-hardened syntax correction. Returns "{}" if no valid brackets are found or the string is blank.
toHtml(@Nullable String response)	String	Extracts clean, plain HTML content from markdown boundaries (```html) or simple tagged strings. Returns empty string "" if blank/null.
getMimeType(String imageUrl)	String	Examines the image URL extension and builds a standard MIME type (e.g., image/jpeg). Defaults to image/png if an extension is missing or unresolvable.
downloadBytes(String imageUrl)	byte[]	Opens a network input stream to the specified image URL and downloads the binary payload into memory.

---

5. Real-World Integration Patterns

5.1 Pattern 1: JSON Structure Synthesis & Schema Mapping

Many service flows require getting a strongly typed Java DTO back from the LLM. In this pattern, the application instructs the model to generate a raw JSON schema, sanitizes the result with LlmUtils.toJson, and converts it using a standard Jackson ObjectMapper.

Here is a simplified, abstract example:

import com.fasterxml.jackson.databind.ObjectMapper;
import io.milesoft.commons.json.ObjectMapperFactory;
import io.milesoft.vertex.services.LlmService;
import io.milesoft.vertex.utils.LlmUtils;
import org.springframework.stereotype.Service;

@Service
public class StructuredAnalysisService {

    private final LlmService llmService;
    private final ObjectMapper objectMapper;

    public StructuredAnalysisService(LlmService llmService) {
        this.llmService = llmService;
        // Use a lenient object mapper to ignore extra fields or minor syntax anomalies
        this.objectMapper = ObjectMapperFactory.lenientObjectMapper();
    }

    public AnalysisResultDto analyzeInputText(String content) {
        String prompt = String.format(
            "Analyze the following content and extract key categories and keywords: '%s'. " +
            "Your output must be structured strictly as a JSON object matching this schema: " +
            "{ \"categories\": [\"string\"], \"keywords\": [\"string\"] }. CONTENT:\n" +
            content
        );

        // 1. Dispatch prompt to Gemini model via our SDK
        String rawLlmResponse = llmService.generateText(prompt);

        // 2. Extract and sanitize JSON block, ignoring conversational pre/postambles
        String sanitizedJson = LlmUtils.toJson(rawLlmResponse);

        // 3. Map safely to a local DTO object
        try {
            return objectMapper.readValue(sanitizedJson, AnalysisResultDto.class);
        } catch (Exception e) {
            throw new IllegalStateException("Failed to parse synthesized analysis data for content", e);
        }
    }
}

5.2 Pattern 2: Multimodal Image Analysis & Insight Extraction

This pattern shows how downstream applications (like AiService) perform visual processing. It feeds a public image URL and a specific text query to the complex multimodal model, extracting structured technical observations.

import io.milesoft.vertex.services.LlmService;
import io.milesoft.vertex.utils.LlmUtils;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Service;

@Slf4j
@Service
public class ConstructionAnalysisService {

    private final LlmService complexLlmService;

    // Inject our complex-reasoning Gemini instance
    public ConstructionAnalysisService(@Qualifier("complexLlmService") LlmService complexLlmService) {
        this.complexLlmService = complexLlmService;
    }

    public void auditSitePhoto(String photoUrl) {
        String prompt = "Perform a safety audit and environmental analysis on this site photo. " +
                        "Identify any visible hazards, safety gear compliance issues, and general weather conditions. " +
                        "Output your findings in HTML format wrapped in an article element.";

        // 1. Pass image URL + prompt directly (LlmService handles binary download and mime formatting)
        String rawResponse = complexLlmService.generateText(photoUrl, prompt);

        // 2. Clean out HTML code blocks for direct safe injection into web pages
        String cleanHtml = LlmUtils.toHtml(rawResponse);

        log.info("Site audit HTML report: {}", cleanHtml);
    }
}

---

6. Troubleshooting & FAQ

Q: Why am I getting "Permission Denied" or application credential errors?

A: Vertex AI requires Google Cloud authentication.

• In GCP Production Environment (Cloud Run, App Engine, GKE): The underlying Google GenAI SDK automatically resolves credentials from the default service account attached to the execution resource.
• In Local Development: You must set your environment credentials:
  • Export the environment variable GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/service-account.json.
  • Alternatively, make sure you are authenticated with the correct active project context by running:

    gcloud auth application-default login
    

Q: Why does Jackson parsing fail on some Gemini responses?

A: LLMs are conversational by nature and often return markdown wrap decorations (e.g., ```json ... ```) or conversational text around the requested JSON payload. Always pass the raw output of llmService.generateText(prompt) through LlmUtils.toJson(...) before parsing. This trims away markdown or conversation and repairs trailing commas, unquoted keys, or malformed quote tokens via the built-in JSON Sanitizer.

Q: What should I use as the "location" parameter?

A: Model availability varies by GCP region. Generally, we recommend deploying your services in standard locations such as us-central1 or us-west1. Check your organization's resource access configuration or consult the Google Cloud Vertex AI documentation for up-to-date regional model listings.