Indexer Core

Indexer Core Skill

CRITICAL RULES

1. Read reference files first. When the user's request matches a topic in the table below, read those files before writing code, proposing architecture, or answering behavioral questions. 2. Treat mode selection as a correctness decision. LogsIndexer and BlockIndexer are not interchangeable. Do not present them as equivalent options with different performance profiles. 3. Default to `IndexerFactory`. For normal library usage, indexers should be configured and built with IndexerFactory, not by manually wiring implementation classes. 4. Treat startup rollback as intentional. It is part of the data-integrity model and reorg recovery workflow, not a bug. 5. Prefer bundled references over ad hoc code spelunking. If you are working inside the indexer-core repository, align with the local repo docs and AGENTS.md. Use source code mainly to confirm implementation details or debug discrepancies.

Scope

Use this skill for indexer-core tasks such as:

• integrating the library into another service
• choosing between LogsIndexer and BlockIndexer
• configuring IndexerFactory, IndexerProcessor, and IndexerRunner
• designing ABI-event, VET-transfer, or business-event indexing setups
• debugging dependency ordering, fast sync, rollback, and reorg behavior
• changing the library itself while preserving documented behavior
• answering migration questions for 7.x to 8.x consumers

Operating Procedure

1. Classify the task

Decide whether the user needs:

• consumer guidance for integrating or configuring the library
• library maintenance for changing indexer-core itself

For consumer guidance, optimize for correct mode selection and integration advice before discussing internals.

For library maintenance, preserve the documented contract unless the task explicitly changes that contract.

2. Read the matching references

Use the table below and load only the files needed for the current request.

3. Clarify the high-risk choices before implementing

Ask before building when any of these are unclear:

• whether the user needs full block access or only decoded events
• whether one indexer must finish a block before another processes it
• whether downstream consumers want raw ABI events or higher-level business events
• whether the task is a behavior change, a docs change, or a debugging task

4. Implement with indexer-core correctness

• build normal indexers through IndexerFactory
• assume repo docs are the authoritative description of public behavior
• keep rollback and reorg semantics intact unless the task explicitly changes them
• do not infer public contracts from a single implementation detail or type signature

5. Verify and deliver

A task is not complete until all applicable gates pass:

1. Targeted verification for the touched behavior 2. Broader tests with ./gradlew test when the change is cross-cutting 3. Formatting with ./gradlew spotlessCheck or ./gradlew spotlessApply when Kotlin code changed 4. Docs consistency when public behavior or examples changed

Reference Files

Read the matching files before doing anything else.

Data Modeling: Monetary and Large-Number Fields

BigDecimal with DECIMAL128 — the Richlist Pattern

When indexing fields that represent token amounts, stakes, balances, or any large integer values from smart contract events, follow the richlist pattern: store as BigDecimal with MongoDB's DECIMAL128 for precision and querying, expose as BigInteger in JSON responses for correct serialization.

Why this split?

• MongoDB side (`BigDecimal` + `DECIMAL128`): Native numeric type enables aggregation queries ($sum, $avg, comparison operators). No precision loss for values up to 34 significant digits.
• JSON side (`BigInteger`): The existing JacksonConfig serializes BigInteger as quoted strings via ToStringSerializer, preventing scientific notation (1e+21) and JavaScript number overflow. BigDecimal does NOT have this serializer — returning BigDecimal directly produces scientific notation for large values.

Model pattern

Use @JsonIgnore on the BigDecimal storage field and expose a BigInteger getter via @JsonProperty:

import com.fasterxml.jackson.annotation.JsonIgnore
import com.fasterxml.jackson.annotation.JsonProperty
import java.math.BigDecimal
import java.math.BigInteger
import org.springframework.data.mongodb.core.mapping.Field
import org.springframework.data.mongodb.core.mapping.FieldType

@Document(collection = "my_collection")
data class MyEntity(
    // Storage field — BigDecimal for MongoDB DECIMAL128
    @JsonIgnore @Field(targetType = FieldType.DECIMAL128) val stake: BigDecimal,
) : VersionedDocument {

    // JSON field — BigInteger for safe serialization as quoted string
    @get:JsonProperty("stake")
    val stakeValue: BigInteger
        get() = stake.toBigInteger()
}

Real-world example — B3TR Richlist

B3trBalance stores balances as BigDecimal/DECIMAL128. The richlist service converts to BigInteger when building the response DTO:

// Model (BigDecimal for storage)
@Field(targetType = FieldType.DECIMAL128) var totalBalance: BigDecimal

// Response DTO (BigInteger for JSON)
data class B3trRichlistItem(val address: String, val balance: BigInteger, val rank: Long)

// Service converts
B3trRichlistItem(
    address = doc.address,
    balance = balanceForScope(doc, scope).toBigIntegerExact(),
    rank = startRank + index,
)

Service pattern — parsing event params

Event parameters arrive as strings. Parse to BigDecimal with null-safe fallback:

private fun String?.toBigDecimalOrZero(): BigDecimal =
    this?.toBigDecimalOrNull() ?: BigDecimal.ZERO

// Usage
val stake = ev.params.getAsString("stakeAmount").toBigDecimalOrZero()

Service pattern — arithmetic

Use BigDecimal operators (Kotlin overloads +, -, comparison):

val newTotal = nav.totalDelegated + amount
val remaining = maxOf(BigDecimal.ZERO, nav.stake - slashAmount)

Overview/aggregate endpoints

For aggregate DTOs, use BigInteger directly and let JacksonConfig handle serialization:

data class NavigatorOverview(
    val totalStaked: BigInteger,   // serialized as "125000" (quoted string)
    val totalDelegated: BigInteger,
)

// In service — aggregate BigDecimal, convert at the end
var totalStaked = BigDecimal.ZERO
for (nav in navigators) { totalStaked += nav.stake }
return NavigatorOverview(totalStaked = totalStaked.toBigInteger(), ...)

Common mistakes

1. Returning `BigDecimal` directly in JSON → produces 1e+21 for large values (no global ToStringSerializer) 2. Using `String` instead of `DECIMAL128` → MongoDB can't run numeric queries or aggregations 3. Using `BigDecimal.toString()` → can produce scientific notation; use toBigInteger() for response fields

Event Pipeline

Use this reference when the user needs event decoding or business-event guidance.

Event Stack

The library's event pipeline is built around CombinedEventProcessor and can include:

• ABI event decoding from classpath JSON resources
• synthetic VET transfer events
• business events derived from decoded events

In normal usage, this is configured through IndexerFactory.

ABI Events

ABI files are loaded from the classpath, not arbitrary filesystem paths.

Useful factory options:

• abis(basePath)
• abiEventNames(...)
• abiContracts(...)

If a requested ABI event or function name is missing, loading fails rather than silently ignoring the mismatch.

Thor-Side Filtering

In log-based mode, filter remote log volume before decoding with:

• eventCriteriaSet(...)
• transferCriteriaSet(...)

This is the most efficient way to narrow log queries.

VET Transfers

Native VET transfers are represented as synthetic IndexedEvent values with:

• eventType = "VET_TRANSFER"
• params for from, to, and amount

They can be enabled explicitly with includeVetTransfers().

If a business-event definition depends on VET_TRANSFER, transfer decoding is enabled automatically.

Business Events

Business events are higher-level actions derived from one or more decoded events in the same transaction.

Use them when downstream consumers care about semantic actions such as staking, claims, swaps, or composite flows.

Useful factory options:

• businessEvents(basePath, abiBasePath)
• businessEventNames(...)
• businessEventContracts(...)
• businessEventSubstitutionParams(...)

ABI vs Business Events

Prefer raw ABI events when:

• every decoded event matters individually
• there is no stable semantic grouping

Prefer business events when:

• downstream consumers want domain actions instead of raw logs

When both are configured, ABI events covered by a business event for the same transaction and clause are removed from the final output to avoid double-reporting.

Maintenance And Migration

Use this reference when changing the library or helping users migrate.

Source Of Truth

When working in the indexer-core repository:

1. read README.md 2. read docs/README.md 3. read the task-specific guide in docs/

Treat the repo markdown docs as the authoritative public contract.

Verification Expectations

For code changes:

• run targeted tests for the touched behavior
• run ./gradlew test when the change is cross-cutting
• run ./gradlew spotlessCheck or ./gradlew spotlessApply when Kotlin code changed

Be explicit if full verification was not run.

Agent Guidance For The Repo

If the repository contains an AGENTS.md, align with it before making behavioral claims. That file should be treated as the repository's canonical agent briefing.

8.0.0 Migration Points

The main 7.x to 8.x changes called out by the repo docs are:

• IndexingResult.Normal was renamed to IndexingResult.BlockResult
• IndexingResult.EventsOnly was renamed to IndexingResult.LogResult
• pruner functionality was removed

When helping users migrate, focus on:

• updating when branches and type checks
• removing pruner-related configuration and code
• verifying status handling against the current enum values

Maintenance Guardrails

• preserve rollback and reorg semantics unless the task explicitly changes them
• do not present internal implementation details as stable public API unless the docs support that
• update public docs when behavior or usage guidance changes

Mode Selection

Use this reference when the user needs help choosing or debugging indexer mode.

Core Rule

LogsIndexer and BlockIndexer are not interchangeable. Choose based on data and execution requirements, not just speed.

Choose LogsIndexer When

Use the default factory-built log mode when the consumer:

• only needs decoded ABI events, business events, or VET transfers
• wants the fastest catch-up path
• does not need full Block contents
• does not require same-block dependency ordering with another indexer

LogsIndexer fast-syncs by querying Thor log endpoints over block ranges and emitting IndexingResult.LogResult.

Choose BlockIndexer When

Use block mode when the consumer needs:

• full block contents
• reverted transaction visibility
• gas or fee metadata that comes from full block processing
• clause inspection via callDataClauses(...)
• same-block dependency ordering through dependsOn(...)

Block mode emits IndexingResult.BlockResult.

Factory Triggers

IndexerFactory.build() returns block-based execution semantics when either of these is set:

• includeFullBlock()
• dependsOn(...)

Otherwise it defaults to log-based execution.

Dependency Implications

If one indexer must finish block N before another handles block N, use dependsOn(...).

That is not a minor optimization hint. It changes how the runner coordinates execution and requires block-based processing.

Common Mistakes To Avoid

• saying LogsIndexer and BlockIndexer only differ in performance
• recommending includeFullBlock() without a concrete need for full block data
• adding dependsOn(...) casually when the user only needs eventual consistency
• forgetting that processors may need to handle different IndexingResult shapes depending on configuration

Runtime Model

Use this reference when the user needs to understand how indexer-core behaves at runtime.

Core Types

• IndexerProcessor: consumer-owned persistence boundary
• IndexerFactory: configures and builds indexers
• IndexerRunner: initialises, fast-syncs when possible, coordinates dependencies, and keeps indexers running
• Indexer: runtime interface implemented by LogsIndexer and BlockIndexer

In normal usage, the consumer implements IndexerProcessor, builds indexers with IndexerFactory, and runs them through IndexerRunner.launch(...).

Processor Responsibilities

IndexerProcessor is responsible for:

• returning the last successfully persisted block with getLastSyncedBlock()
• rolling back persisted state with rollback(blockNumber)
• processing either IndexingResult.LogResult or IndexingResult.BlockResult

The processor is the application boundary. Persistence logic belongs there, not inside the library.

Lifecycle

Typical lifecycle:

1. the runtime queries getLastSyncedBlock() 2. it rolls back from that block if needed 3. the indexer moves to INITIALISED 4. LogsIndexer may fast-sync to the latest finalized block 5. steady-state block processing begins

Status values:

• NOT_INITIALISED
• INITIALISED
• FAST_SYNCING
• SYNCING
• FULLY_SYNCED
• SHUT_DOWN

Startup Rollback

Rollback on startup is intentional.

The library reprocesses from a known safe point so downstream state stays correct after restarts, partial writes, or reorg recovery. Do not describe this as accidental or suspicious behavior.

Reorg Handling

Reorg handling is built into the runtime:

• block-based execution compares expected parent linkage against the canonical chain
• on mismatch, the runtime throws a ReorgException
• the runner re-initialises and resumes from rolled-back state

Consumers are expected to provide deterministic rollback behavior in IndexerProcessor.

Dependency Ordering

dependsOn(...) means more than "run this first eventually."

It tells the runtime that one indexer must finish a given block before another indexer processes that same block. This changes execution semantics and forces block-based coordination.

Runner Behavior

IndexerRunner does all of the following:

• initialises configured indexers
• fast-syncs fast-syncable indexers
• may run independent non-fast-syncable indexers while that fast sync is happening
• forms dependency-aware execution groups
• processes indexers in topological order within each group
• retries failures until success or cancellation

When explaining throughput, keep correctness first. Dependency chains constrain same-block ordering.