Milesoft Firestore Extension (firestore-ext)

1. Overview & Purpose

The Firestore Extension (firestore-ext) is a unified, highly optimized, and robust facade library built on top of the official Google Cloud Firestore Java SDK. Designed specifically for the Milesoft ecosystem, it enforces enterprise-grade architectural consistency, standardized CRUD behaviors, seamless integration with Machine Learning vector embeddings, and highly resilient paging iterator patterns across downstream applications and services.

Key Benefits & Core Capabilities:

  • Type-Safe Data Access Layer (FirestoreDao): Standardizes standard database operations, abstracts document reference mapping, and reduces boilerplate code when writing custom persistence layers.
  • Automated Vector Embedding Integration: Seamlessly coordinates with an EmbeddingProvider to serialize target entity fields as a unified text surface, generate ML-powered embeddings, and store them as Native Firestore Vectors (FieldValue.vector(...)) during entity saves.
  • Resilient Pagination & Streaming: Provides high-throughput cursor pagination and out-of-the-box support for streaming massive datasets using lazy-loaded, paged background iterators without memory overhead.
  • Write Anomaly & Frequency Detection: Utilizes thread-local diagnostic contexts to track high-frequency single-document write anomalies in real-time, proactively encouraging developers to leverage batched bulk APIs.
  • Automatic Query Partitioning (IN-Clause Expansion): Seamlessly overcomes Firestore’s native 30-item constraint on in / whereIn queries by automatically dividing large collections of IDs into concurrent batch sub-queries and coalescing the results.
  • Safe Batch Operations: Built-in wrapper support for Google Cloud's high-efficiency BulkWriter for massive inserts, updates, and deletes with customizable paging hooks.

2. Installation & Dependency Configuration

The firestore-ext library resides inside our secure Google Cloud Artifact Registry repository.

To pull this package into your downstream Java application, configure your build file as follows:

2.1 Gradle Setup (build.gradle)

Apply the Google Cloud Artifact Registry plugin and register our secure Maven repository:

plugins {
    id "com.google.cloud.artifactregistry.gradle-plugin" version "2.2.5"
    id "java"
}

repositories {
    mavenLocal()
    mavenCentral()
    maven {
        url "artifactregistry://us-west2-maven.pkg.dev/milesoft-repo/repo-maven"
    }
}

dependencies {
    // Required Milesoft foundational libraries
    implementation "io.milesoft:commons:2.0.1"
    
    // Milesoft Firestore Extension
    implementation "io.milesoft:firestore-ext:2.0.1"
}

3. Core Entities & Structures

The library establishes strict type-safety boundaries and standardizes object state.

3.1 FirestoreEntity

Every domain model persisted via the extension must implement the FirestoreEntity interface. This ensures all documents share uniform identifiers and auditing timestamps. The interface is annotated to prevent computed vector embeddings from polluting standard client serialization blocks:

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import jakarta.annotation.Nullable;
import java.time.ZonedDateTime;

@JsonIgnoreProperties(value = {"embedding"}, ignoreUnknown = true)
public interface FirestoreEntity {

    void setId(String setId);

    void setCreated(ZonedDateTime created);

    void setUpdated(ZonedDateTime updated);

    @Nullable
    @JsonProperty(access = JsonProperty.Access.WRITE_ONLY)
    String getId();

    @Nullable
    ZonedDateTime getCreated();

    @Nullable
    ZonedDateTime getUpdated();
}

3.2 PagedResults<E, C>

An immutable generic container returned by paginated search queries. It encapsulates the resulting list of entities for the current page along with the strongly typed cursor token (C) corresponding to the last evaluated document:

import java.util.List;
import lombok.Builder;
import lombok.Getter;

@Getter
@Builder(builderClassName = "Builder")
public class PagedResults<E, C> {
    private final List<E> entities;
    private final C cursor; // Strongly typed cursor (e.g., String, ZonedDateTime)
}

3.3 Anomaly Detection Context

To prevent catastrophic database performance degradation, the SDK monitors write frequencies on a thread-local basis. If a thread performs numerous single-document operations (e.g., save or delete) within a one-second window, the SDK logs an actionable alert encouraging the use of bulk APIs:

[WARN] High frequency of single-document writes detected on thread main for collection accounts. 
Consider using bulkSave() or bulkDelete() for better performance and reliability.

4. Implementing and Configuring a Custom DAO

To manage a Firestore collection, create a Spring-managed component that extends FirestoreDao<E>.

4.1 Required Base Implementation

Your custom DAO subclass must implement two simple metadata hooks:

  • getCollectionType(): The physical name of the target Firestore collection.
  • getEntityClass(): The exact class token representing the mapped entity.

4.2 Standard DAO Example

The following is an implementation of a cache-enabled, Spring @Component DAO that leverages Spring's Clock and caching layers while extending FirestoreDao:

import static com.google.cloud.firestore.FieldPath.documentId;
import static com.google.common.base.Preconditions.checkArgument;
import static io.milesoft.commons.constants.Limits.LARGE;
import static io.milesoft.commons.constants.Limits.SMALL;

import com.google.cloud.firestore.Query;
import com.yourcompany.myapp.firestore.entity.Animal;
import io.milesoft.firestore.dao.FirestoreDao;
import io.milesoft.firestore.domain.PagedResults;
import jakarta.annotation.Nullable;
import java.time.Clock;
import java.util.List;
import java.util.Optional;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cache.annotation.CacheConfig;
import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Component;

@Component
@CacheConfig(cacheNames = "animals")
public class AnimalDao extends FirestoreDao<Animal> {

    @Autowired
    public AnimalDao(Clock clock) {
        super(clock); // Automatically initializes Firestore using the default GCP service configuration
    }

    @Override
    protected String getCollectionType() {
        return "animals";
    }

    @Override
    protected Class<Animal> getEntityClass() {
        return Animal.class;
    }

    // Leveraging FirestoreDao protected helper queries
    @Cacheable(key = "'list_' + #cursor + '_' + #limit")
    public PagedResults<Animal, String> findAnimals(@Nullable String cursor, int limit) {
        checkArgument(limit >= SMALL && limit <= LARGE, "Invalid limit bounds");

        Query query = reference
            .orderBy(documentId(), Query.Direction.ASCENDING)
            .limit(limit);

        if (StringUtils.isNotBlank(cursor)) {
            query = query.startAfter(cursor);
        }

        // Standardized cursor retrieval using method reference
        return toPagedResults(query, Animal::getId);
    }
}

5. API Reference Guide (Base Operations)

FirestoreDao provides a rich set of built-in methods. These fall into several categories:

5.1 Basic CRUD Operations

Method Signature Return Type Description
save(E entity) E Saves an entity. If id is blank, generates a new Firestore Document ID and sets both created and updated. If id is populated, performs a full write/overwrite and refreshes updated.
load(String id) Optional<E> Attempts to load a document by its ID and maps it to the target entity class. Returns Optional.empty() if the document does not exist.
exists(String id) boolean Returns true if a document exists matching the provided ID.
delete(String id) void Deletes a specific document by its ID.
updateSelective(String id, Map<String, Object> updates) void Partially updates specified fields on a document. Automatically sets and formats the updated timestamp.

5.2 Bulk & Mass Deletion Operations

Method Signature Return Type Description
bulkSave(List<E> entities) void Highly optimized mass insert/update leveraging a Firestore BulkWriter. Splices inputs automatically into separate batched runs of 500 records. Generates vector embeddings for entities if configured.
bulkDelete(List<String> ids) void Efficiently deletes multiple document IDs using a native BulkWriter.
deleteAll() void Automatically deletes all documents inside the current collection using paged chunk-deletes to avoid timeout or memory failures.
deleteAll(Consumer<List<E>> callback) void Identical to deleteAll(), but invokes a caller-defined callback with each loaded batch of entities immediately prior to execution (extremely useful for cache evictions).
bulkDelete(Query query) void Runs a query, extracts matching document IDs, and deletes them chunk-by-chunk using a BulkWriter.
bulkDelete(Query query, Consumer<List<E>> callback) void Identical to query-based deletion, but accepts a consumer callback containing mapped entities for each chunk before triggering their deletion.

6. Query and Collection Retrieval Utilities

Writing custom queries on raw Firestore SDK objects is verbose. FirestoreDao provides a suite of protected helper methods designed to cleanly map results.

6.1 Protected Query Handlers

These methods are designed to be used inside your custom DAO implementation:

Method Return Type Description
toOptional(Query query) Optional<E> Fetches the first result matching the query and wraps it.
toList(Query query) List<E> Executes the query and returns a fully mapped, type-safe list.
toIds(Query query) List<String> Lightweight query that selects and returns only matching Document IDs.
toCount(Query query) int Executes a high-performance, lightweight Firestore aggregation count query.
toPagedResults(Query query, Function<E, C> cursorGetter) PagedResults<E, C> Executes a limit-constrained query, parses the list of entities, and applies the cursorGetter function to extract the next page token.
findByIds(Collection<String> ids) List<E> Performs high-speed parallel chunk lookups using firestore.getAll(refs) in chunks of 1000 IDs, filter-filtering non-existent matches.

6.2 Bypassing the 30-Item IN Constraint

Native Firestore restricts whereIn queries to a maximum of 30 arguments, which often forces developers to write complex looping logic. The Firestore Extension abstracts this away entirely.

If you have a collection of IDs or keys of arbitrary size and want to query them in one shot, utilize the concurrent in-clause partition engines:

  • toListWithInClause(Query baseQuery, String field, Collection<?> inValues)
  • toListWithInClause(Query baseQuery, FieldPath fieldPath, Collection<?> inValues)

Implementation Pattern:

Under the hood, these methods segment your inValues into batches of 30 (controlled by MAX_IN_QUERY_SIZE), triggers up to 10 concurrent parallel queries (controlled by MAX_CONCURRENT_QUERIES) using ApiFutures, waits for their parallel execution, and joins the results into a single consolidated, deduplicated List<E>.

7. Stream Processing & Paging Iterator

For large-scale offline processing, daily cron executions, or data exports, loading entire collections into heap memory is extremely risky. FirestoreDao features a custom-engineered PagingIterator that lets you stream an entire query's result set lazily.

7.1 Streaming APIs

Method Return Type Description
streamAll() Stream<E> Streams all documents inside the collection using cursor pagination in background batches.
stream(Query query) Stream<E> Streams a specific custom query's results lazily using a background-paging iterator in batches of 500 documents.

Code Usage Example:

You can process a massive collection utilizing Java's Stream pipeline. Memory usage remains constant because documents are loaded, mapped, processed, and garbage-collected in batches of 500:

// Streams and processes hundreds of thousands of notifications without memory exhaustion
try (Stream<Notification> stream = notificationDao.streamAll()) {
    stream.filter(notification -> "FAILED".equals(notification.getStatus()))
          .forEach(failedNotify -> {
              log.info("Processing failed notification ID: {}", failedNotify.getId());
              // Execute retry rules...
          });
}

8. Machine Learning & Vector Search Integration

A core strength of firestore-ext is its native, hands-free support for Vector Searches (Retrieval-Augmented Generation / RAG) utilizing Firestore’s high-density index capabilities.

8.1 The Core Integration Mechanics

  1. EmbeddingProvider: Implement this simple functional interface to hook up your preferred AI LLM or embedding model (e.g., Google Vertex AI, OpenAI, HuggingFace):
    public interface EmbeddingProvider {
    float[] embed(String text);
    
    default List<float[]> embed(List<String> texts) {
        return texts.stream().map(this::embed).toList();
    }
}
  2. getEmbeddingFields(): Override this method inside your custom DAO to define which property keys in your entity contain the textual knowledge.
  3. Automatic Serialization: When saving documents via save() or bulkSave(), the DAO checks your registered EmbeddingProvider. It serializes the designated fields into a structured JSON string "text surface", generates a floating-point vector, translates it to double-precision format, and inserts it under the "embedding" document field as a native vector.

8.2 Real-World Vector Search Implementation

Here is how you implement a fully operational Vector Search DAO (KnowledgeDao) using Spring AI's EmbeddingModel and VertexEmbeddingProvider:

import static com.google.cloud.firestore.VectorQuery.DistanceMeasure.COSINE;
import static io.milesoft.firestore.constants.FirestoreConstants.FIELD_EMBEDDING;
import static io.milesoft.firestore.utils.FirestoreUtils.formatForQuery;

import com.google.cloud.firestore.VectorQuery;
import com.google.cloud.firestore.VectorQueryOptions;
import io.milesoft.firestore.dao.FirestoreDao;
import io.milesoft.firestore.utils.EmbeddingProvider;
import java.time.Clock;
import java.util.List;
import java.util.Set;
import org.springframework.stereotype.Component;

@Component
public class KnowledgeDao extends FirestoreDao<Knowledge> {

    public KnowledgeDao(Clock clock, EmbeddingProvider vertexEmbeddingProvider) {
        super(clock, vertexEmbeddingProvider); // Passes the custom embedding generator to the facade
    }

    @Override
    protected String getCollectionType() {
        return "knowledge";
    }

    @Override
    protected Class<Knowledge> getEntityClass() {
        return Knowledge.class;
    }

    // Automatically hooks up text surface serialization
    @Override
    protected Set<String> getEmbeddingFields() {
        return Set.of("textSurface", "title", "category");
    }

    // High-performance Vector Similarity Search
    public List<Knowledge> findSimilarKnowledge(float[] queryVector, double similarityThreshold, int limit) {
        final VectorQueryOptions options = VectorQueryOptions.newBuilder()
            .setDistanceThreshold(1.0 - similarityThreshold) // Translate cosine similarity to distance
            .build();

        // Performs vector nearest-neighbor search
        final VectorQuery vectorQuery = reference
            .findNearest(FIELD_EMBEDDING, formatForQuery(queryVector), limit, COSINE, options);

        // Map results back to entities
        return toList(vectorQuery);
    }
}

9. Helper Utilities

The extension package ships with valuable standalone helper classes that streamline date formatting, key composition, and paginated background processes.

9.1 CursorUtils

For applications that need to iterate through complete collections but cannot extend FirestoreDao or need a high-level iterator interface, CursorUtils provides a set of static pagination orchestrators:

  • runForAll(loader, consumer) / parallelRunForAll(loader, consumer): Loops through all pages using a cursor and applies the consumer serial-by-serial or across a parallel stream.
  • loadAll(loader): Loops through all cursor pages and returns a single, unified, unmodifiable List<T>.

Code Example:

// Accumulate all elements across thousands of paginated database calls in one line
List<Account> allAccounts = CursorUtils.loadAll(
    (cursor, limit) -> accountDao.findAccounts(cursor, limit)
);

9.2 FirestoreUtils

Contains standard helpers for safe composite key generation, date conversion, and sanitization:

Utility Method Return Type Description
compositeKey(part1, part2, ...) String Joins parameters into a standardized composite key string using a colon separator (":").
toParts(compositeKey) List<String> Splits a composite key back into its original parts.
formatForQuery(ZonedDateTime timestamp) String Formats timestamps to ISO offset date-time aligned with UTC zone.
formatForQuery(LocalDate date) String Formats local dates to ISO date string (YYYY-MM-DD).
formatForQuery(float[] embedding) double[] Converts a float array to double precision and clips length to MAX_EMBEDDING_DIMENSIONS (2048) for secure Vector storing.
makeCaseInsensitive(String str) String Sanitizes search filters by lowering case and trimming spaces.
removeNonNumeric(String str) String Strips all non-numeric characters from a string (highly useful for cleaning phone numbers).