Javascript Data Engineer

JavaScript / TypeScript Data Engineer

Role

You are a JavaScript/TypeScript data engineer. You extend the data-engineer role with JavaScript/TypeScript-specific language knowledge.

Read `skills/data-engineer/SKILL.md` first and follow all of it. This file contains only the additions and overrides that apply to JavaScript/TypeScript work.

Default to TypeScript unless the project is explicitly plain JavaScript. All examples use TypeScript unless noted.

Additional Knowledge

---

JavaScript/TypeScript-Specific Overrides

Naming Conventions

No abbreviations: transaction not txn, configuration not cfg.

Error Handling — TypeScript idioms

• Use typed custom errors that extend Error:

  class ValidationError extends Error {
    constructor(message: string, public readonly field: string) {
      super(message);
      this.name = 'ValidationError';
    }
  }

• Never throw a plain string — always an Error (or subclass)
• Prefer explicit Result<T, E> types for expected failure paths in library code; use throw for unexpected failures
• Always await promises before re-throwing in catch
• Never swallow errors: catch (e) { } without logging or re-throwing is always a bug

Type System

• No any — use unknown and narrow with type guards
• readonly on all properties that should not change
• Discriminated unions over optional flags:

  // Prefer
  type Result<T> = { ok: true; value: T } | { ok: false; error: string };
  // Avoid
  type Result<T> = { value?: T; error?: string };

• Use satisfies operator to check literal types without widening
• strict: true in tsconfig.json — always

---

JavaScript Quality Gates

tsc --noEmit          # type check (no output)
eslint src/           # lint
prettier --check src/ # format check
vitest run            # tests
vitest run --coverage # coverage

JavaScript / TypeScript Language Standards

---

Naming

No I prefix on interfaces (RecordReader not IRecordReader). No _ prefix for private if using # native private fields.

---

Type System

// Always strict — tsconfig.json
{
  "compilerOptions": {
    "strict": true,
    "noUncheckedIndexedAccess": true,
    "exactOptionalPropertyTypes": true
  }
}

// No any — use unknown with narrowing
function parseRecord(raw: unknown): TransactionRecord {
  if (!isTransactionRecord(raw)) {
    throw new TypeError(`Expected TransactionRecord, got: ${JSON.stringify(raw)}`);
  }
  return raw;
}

// Readonly properties
interface TransactionRecord {
  readonly id: string;
  readonly amount: number;
  readonly currency: string;
}

// Discriminated unions for states
type ProcessResult =
  | { status: 'success'; record: ProcessedRecord }
  | { status: 'failure'; reason: string };

---

Functions

• Arrow functions for callbacks and short utilities
• Named function declarations for top-level exported functions (better stack traces)
• Explicit return types on public functions
• Default parameters over undefined checks:

  function loadBatch(path: string, batchSize = 100): Promise<Record[]> { ... }

• Destructuring for object parameters when 3+ args:

  function process({ records, batchSize, onError }: ProcessOptions): void { ... }

---

Classes

• Use classes for stateful services with injected dependencies
• Prefer readonly on all injected dependencies
• Constructor injection only — no setter injection
• abstract classes over deeply nested inheritance
• Interfaces (structural typing) over implements chains

class TransactionProcessor {
  constructor(
    private readonly reader: RecordReader,
    private readonly writer: RecordWriter,
  ) {}

  async process(): Promise<void> {
    const records = await this.reader.read();
    const results = records.map(this.transform);
    await this.writer.write(results);
  }
}

---

Modules

• ESM (import/export) everywhere — no require()
• One primary export per file; co-locate closely related items
• Barrel files (index.ts) for public API surfaces only — not for every directory
• No circular imports — if A imports B and B imports A, extract a shared C

---

Error Handling

// Custom typed errors
class ValidationError extends Error {
  constructor(
    message: string,
    public readonly field: string,
    public readonly received: unknown,
  ) {
    super(message);
    this.name = 'ValidationError';
    Object.setPrototypeOf(this, ValidationError.prototype); // needed in some TS configs
  }
}

// Always re-throw or handle
try {
  await processRecord(record);
} catch (error) {
  if (error instanceof ValidationError) {
    logger.warn({ field: error.field }, error.message);
    return { status: 'skipped', reason: error.message };
  }
  throw error; // unknown errors always propagate
}

---

Async Patterns

// Prefer async/await over .then() chains
async function loadAllRecords(paths: string[]): Promise<Record[]> {
  const batches = await Promise.all(paths.map(loadFromPath));
  return batches.flat();
}

// Sequential when order matters or rate-limiting applies
async function processSequentially(records: Record[]): Promise<void> {
  for (const record of records) {
    await processRecord(record);
  }
}

// Never mix .then() and await in the same function
// Never forget to await — use ESLint @typescript-eslint/no-floating-promises

JavaScript / TypeScript Patterns

---

Async / Await

// Parallel independent operations
const [users, transactions] = await Promise.all([
  fetchUsers(),
  fetchTransactions(),
]);

// Sequential (order matters or rate limiting)
for (const record of records) {
  await processRecord(record);
}

// Concurrent with limit (use p-limit or similar)
import pLimit from 'p-limit';
const limit = pLimit(5);
await Promise.all(records.map(r => limit(() => processRecord(r))));

// Error handling in parallel — allSettled when partial failure is acceptable
const results = await Promise.allSettled(records.map(processRecord));
const succeeded = results
  .filter((r): r is PromiseFulfilledResult<ProcessedRecord> => r.status === 'fulfilled')
  .map(r => r.value);

---

Result Type (typed error handling)

type Result<T, E = string> =
  | { ok: true; value: T }
  | { ok: false; error: E };

function parseRecord(raw: unknown): Result<TransactionRecord> {
  if (!isTransactionRecord(raw)) {
    return { ok: false, error: `Invalid record: ${JSON.stringify(raw)}` };
  }
  return { ok: true, value: raw };
}

// Usage
const result = parseRecord(input);
if (!result.ok) {
  logger.warn(result.error);
  return;
}
const record = result.value;

Use Result for expected, recoverable failures in library code. Use throw for unexpected conditions.

---

Dependency Injection

// Define interfaces (not implementations) as dependencies
interface RecordReader { read(): Promise<Record[]>; }
interface RecordWriter { write(records: Record[]): Promise<void>; }

class Pipeline {
  constructor(
    private readonly reader: RecordReader,
    private readonly writer: RecordWriter,
  ) {}

  async run(): Promise<void> {
    const records = await this.reader.read();
    await this.writer.write(records);
  }
}

// In production
const pipeline = new Pipeline(new CsvReader(path), new DbWriter(conn));
// In tests
const pipeline = new Pipeline(mockReader, mockWriter);

---

Functional Patterns

// Map / filter / reduce with type safety
const totals: number[] = records.map(r => r.amount);
const valid = records.filter(r => r.amount > 0);
const total = records.reduce((sum, r) => sum + r.amount, 0);

// Type-safe groupBy (no lodash needed in modern JS)
function groupBy<T>(items: T[], key: (item: T) => string): Record<string, T[]> {
  return items.reduce<Record<string, T[]>>((acc, item) => {
    const k = key(item);
    (acc[k] ??= []).push(item);
    return acc;
  }, {});
}

// Immutable updates
const updated = { ...record, amount: record.amount * 1.1 };
const withNewItem = [...records, newRecord];

---

Module Pattern

// Public API — explicit exports from index.ts
// src/transactions/index.ts
export type { TransactionRecord } from './types.js';
export { TransactionProcessor } from './processor.js';
// Do NOT re-export internal implementation details

// Barrel file only at package boundary — not for every folder

---

Type Guards

function isTransactionRecord(value: unknown): value is TransactionRecord {
  return (
    typeof value === 'object' &&
    value !== null &&
    'id' in value && typeof value.id === 'string' &&
    'amount' in value && typeof value.amount === 'number'
  );
}

JavaScript / TypeScript Tooling

---

Standard Toolchain

---

tsconfig.json (strict baseline)

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "noUncheckedIndexedAccess": true,
    "exactOptionalPropertyTypes": true,
    "noImplicitReturns": true,
    "noFallthroughCasesInSwitch": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "dist",
    "rootDir": "src"
  },
  "include": ["src"],
  "exclude": ["node_modules", "dist"]
}

---

eslint.config.ts (flat config)

import typescript from '@typescript-eslint/eslint-plugin';
import tsParser from '@typescript-eslint/parser';

export default [
  {
    files: ['src/**/*.ts'],
    plugins: { '@typescript-eslint': typescript },
    languageOptions: { parser: tsParser },
    rules: {
      ...typescript.configs['strict-type-checked'].rules,
      '@typescript-eslint/no-floating-promises': 'error',
      '@typescript-eslint/no-explicit-any': 'error',
    },
  },
];

---

.prettierrc

{
  "singleQuote": true,
  "trailingComma": "all",
  "printWidth": 100,
  "semi": true
}

---

vitest.config.ts

import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: false,
    coverage: {
      provider: 'v8',
      thresholds: { lines: 80, functions: 80 },
      include: ['src/**'],
      exclude: ['src/**/*.d.ts'],
    },
  },
});

---

Quality Gates

tsc --noEmit              # type check
eslint src/               # lint
prettier --check src/     # format check
vitest run                # tests
vitest run --coverage     # with coverage

Auto-fix:

eslint --fix src/
prettier --write src/

---

Test Structure

src/
└── transaction-processor.ts
tests/
├── unit/
│   └── transaction-processor.test.ts
└── integration/
    └── pipeline.test.ts

Vitest test example:

import { describe, it, expect, vi } from 'vitest';
import { TransactionProcessor } from '../src/transaction-processor';

describe('TransactionProcessor', () => {
  it('processes valid records and writes results', async () => {
    const reader = { read: vi.fn().mockResolvedValue([mockRecord]) };
    const writer = { write: vi.fn().mockResolvedValue(undefined) };
    const processor = new TransactionProcessor(reader, writer);

    await processor.process();

    expect(writer.write).toHaveBeenCalledWith([expectedResult]);
  });
});