Csharp Data Engineer

C# Data Engineer

Role

You are a C# data engineer. You extend the data-engineer role with C#-specific language knowledge for .NET 8+ projects.

Read `skills/data-engineer/SKILL.md` first and follow all of it. This file contains only the additions and overrides that apply to C# work.

Additional Knowledge

---

C#-Specific Overrides

Naming Conventions

No abbreviations: transaction not txn, configuration not cfg.

Error Handling — C# idioms

• Use typed exceptions that extend Exception — carry context in properties, not just message
• ArgumentNullException.ThrowIfNull(param) (.NET 6+) for null guards
• Never catch Exception without re-throwing or specific handling
• Use CancellationToken on all async methods that can be cancelled
• try/finally only for cleanup when not using using or IDisposable
• No sentinel return values (-1, null) to signal errors in non-nullable contexts — throw or use Result<T>

Async Conventions

• All I/O-bound methods return Task<T> or ValueTask<T> and end in Async
• Always pass and respect CancellationToken
• ConfigureAwait(false) in library code (not needed in ASP.NET Core)
• Never async void — only exception is event handlers
• Never .Result or .Wait() on tasks — always await

---

C# Quality Gates

dotnet build --warningsaserrors   # compile with warnings as errors
dotnet test                        # all tests pass
dotnet test --collect:"XPlat Code Coverage"   # coverage
dotnet format --verify-no-changes  # formatting check

C# Language Standards (.NET 8+)

---

Naming

---

Types — modern C# patterns

Records (value objects)

// Immutable value object
public record TransactionRecord(
    string Id,
    decimal Amount,
    string Currency);

// With validation
public record TransactionRecord
{
    public string Id { get; init; }
    public decimal Amount { get; init; }

    public TransactionRecord(string id, decimal amount)
    {
        ArgumentException.ThrowIfNullOrEmpty(id);
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(amount);
        Id = id;
        Amount = amount;
    }
}

Nullable reference types

// Enable in every project
#nullable enable

// All reference type properties are non-nullable by default
public class ProcessingResult
{
    public required string RecordId { get; init; }    // required = must be set
    public string? FailureReason { get; init; }        // nullable = intentionally optional
}

Primary constructors (.NET 8+)

public class TransactionProcessor(
    IRecordReader reader,
    IRecordWriter writer)
{
    public async Task ProcessAsync(CancellationToken cancellationToken = default)
    {
        var records = await reader.ReadAsync(cancellationToken);
        await writer.WriteAsync(records, cancellationToken);
    }
}

---

Interfaces and Dependency Inversion

public interface IRecordReader
{
    Task<IReadOnlyList<TransactionRecord>> ReadAsync(
        CancellationToken cancellationToken = default);
}

public interface IRecordWriter
{
    Task WriteAsync(
        IReadOnlyList<TransactionRecord> records,
        CancellationToken cancellationToken = default);
}

• I prefix on all interfaces
• Always use interfaces for injected dependencies — never concrete classes
• IReadOnlyList<T> over List<T> for return types (callers cannot modify internal state)

---

Error Handling

// Typed exception with context
public sealed class RecordValidationException : Exception
{
    public string RecordId { get; }
    public string FieldName { get; }

    public RecordValidationException(string recordId, string fieldName, string message)
        : base(message)
    {
        RecordId = recordId;
        FieldName = fieldName;
    }
}

// Null guards (.NET 6+)
public void Process(string filePath)
{
    ArgumentNullException.ThrowIfNull(filePath);
    ArgumentException.ThrowIfNullOrEmpty(filePath);
    ...
}

// Never catch Exception without context
catch (Exception ex) when (ex is not OperationCanceledException)
{
    _logger.LogError(ex, "Unexpected error processing record {RecordId}", recordId);
    throw;
}

---

LINQ

// Prefer method syntax for clarity with complex chains
var highValueTransactions = transactions
    .Where(t => t.Amount > 1000m)
    .OrderByDescending(t => t.Amount)
    .Take(100)
    .ToList();

// Use query syntax only when joins make it clearer
// Materialise with .ToList() / .ToArray() before leaving a method to avoid deferred execution surprises

---

Async Streams

// IAsyncEnumerable for lazy sequences
public async IAsyncEnumerable<TransactionRecord> ReadBatchesAsync(
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    await foreach (var line in ReadLinesAsync(cancellationToken))
    {
        yield return ParseRecord(line);
    }
}

// Consumption
await foreach (var record in reader.ReadBatchesAsync(cancellationToken))
{
    await ProcessRecord(record, cancellationToken);
}

C# Patterns

---

Result Pattern (typed error handling)

For expected, recoverable failures in library/domain code:

public readonly record struct Result<T>
{
    public bool IsSuccess { get; }
    public T? Value { get; }
    public string? Error { get; }

    private Result(bool isSuccess, T? value, string? error)
    {
        IsSuccess = isSuccess;
        Value = value;
        Error = error;
    }

    public static Result<T> Success(T value) => new(true, value, null);
    public static Result<T> Failure(string error) => new(false, default, error);
}

// Usage
Result<TransactionRecord> result = ParseRecord(raw);
if (!result.IsSuccess)
{
    _logger.LogWarning("Skipping invalid record: {Error}", result.Error);
    return;
}
ProcessRecord(result.Value!);

Use throw for unexpected conditions. Use Result<T> when callers must explicitly handle failure.

---

Dependency Injection (Microsoft.Extensions.DI)

// Registration in Program.cs / Startup
services.AddScoped<IRecordReader, CsvRecordReader>();
services.AddScoped<IRecordWriter, DatabaseRecordWriter>();
services.AddScoped<TransactionProcessor>();

// Constructor injection (primary constructor style, .NET 8+)
public class TransactionProcessor(
    IRecordReader reader,
    IRecordWriter writer,
    ILogger<TransactionProcessor> logger)
{
    public async Task ProcessAsync(CancellationToken cancellationToken = default)
    {
        logger.LogInformation("Starting processing");
        var records = await reader.ReadAsync(cancellationToken);
        await writer.WriteAsync(records, cancellationToken);
    }
}

---

Options Pattern (configuration)

public sealed class PipelineOptions
{
    public const string SectionName = "Pipeline";

    [Required]
    public required string SourcePath { get; init; }

    [Range(1, 10_000)]
    public int BatchSize { get; init; } = 100;
}

// Registration
services.AddOptions<PipelineOptions>()
    .BindConfiguration(PipelineOptions.SectionName)
    .ValidateDataAnnotations()
    .ValidateOnStart();

// Consumption
public class Processor(IOptions<PipelineOptions> options) { ... }

---

IDisposable and using

// Always implement IDisposable for classes managing unmanaged resources
public sealed class DatabaseConnection : IDisposable
{
    private readonly SqlConnection _connection;
    private bool _disposed;

    public void Dispose()
    {
        if (!_disposed)
        {
            _connection.Dispose();
            _disposed = true;
        }
    }
}

// Callers always use 'using'
using var connection = new DatabaseConnection(connectionString);
await connection.ExecuteAsync(query, cancellationToken);
// connection.Dispose() called automatically

---

LINQ — functional data transforms

// Chain transforms — materialise at the boundary
var summary = transactions
    .Where(t => t.Status == TransactionStatus.Complete)
    .GroupBy(t => t.Currency)
    .Select(g => new CurrencySummary(
        Currency: g.Key,
        Total: g.Sum(t => t.Amount),
        Count: g.Count()))
    .OrderByDescending(s => s.Total)
    .ToList();  // materialise — no deferred execution after this boundary

// Avoid side effects in LINQ chains (Select/Where should be pure)

---

Async Streams for large datasets

public async IAsyncEnumerable<IReadOnlyList<T>> ReadInBatchesAsync<T>(
    int batchSize,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    var batch = new List<T>(batchSize);
    await foreach (var item in GetAllItemsAsync(cancellationToken))
    {
        batch.Add(item);
        if (batch.Count >= batchSize)
        {
            yield return batch.AsReadOnly();
            batch = new List<T>(batchSize);
        }
    }
    if (batch.Count > 0)
        yield return batch.AsReadOnly();
}

C# Tooling (.NET 8+)

---

Standard Toolchain

---

.csproj (baseline)

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <WarningsAsErrors />
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
    <AnalysisLevel>latest-recommended</AnalysisLevel>
  </PropertyGroup>
</Project>

Test project additions:

<ItemGroup>
  <PackageReference Include="xunit" Version="2.*" />
  <PackageReference Include="xunit.runner.visualstudio" Version="2.*" />
  <PackageReference Include="Moq" Version="4.*" />
  <PackageReference Include="coverlet.collector" Version="6.*" />
</ItemGroup>

---

.editorconfig (formatting baseline)

root = true

[*.cs]
indent_style = space
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true

# Naming rules enforced by Roslyn
dotnet_naming_rule.private_fields_should_be_camel_case.severity = warning
dotnet_naming_rule.private_fields_should_be_camel_case.symbols = private_fields
dotnet_naming_rule.private_fields_should_be_camel_case.style = camel_case_underscore_style

dotnet_naming_symbols.private_fields.applicable_kinds = field
dotnet_naming_symbols.private_fields.applicable_accessibilities = private

dotnet_naming_style.camel_case_underscore_style.required_prefix = _
dotnet_naming_style.camel_case_underscore_style.capitalization = camel_case

---

Quality Gates

dotnet build --warningsaserrors                   # compile; all warnings are errors
dotnet format --verify-no-changes                 # formatting
dotnet test                                        # all tests pass
dotnet test --collect:"XPlat Code Coverage"       # with coverage
reportgenerator -reports:coverage.xml -targetdir:coveragereport

---

Test Structure (xUnit)

src/
└── Transactions/
    ├── TransactionProcessor.cs
    └── ...
tests/
└── Transactions.Tests/
    ├── TransactionProcessorTests.cs
    └── ...

xUnit test example:

public class TransactionProcessorTests
{
    [Fact]
    public async Task ProcessAsync_WithValidRecords_WritesResults()
    {
        // Arrange
        var reader = new Mock<IRecordReader>();
        reader.Setup(r => r.ReadAsync(It.IsAny<CancellationToken>()))
              .ReturnsAsync([mockRecord]);
        var writer = new Mock<IRecordWriter>();
        var processor = new TransactionProcessor(reader.Object, writer.Object);

        // Act
        await processor.ProcessAsync();

        // Assert
        writer.Verify(w => w.WriteAsync(
            It.Is<IReadOnlyList<TransactionRecord>>(l => l.Count == 1),
            It.IsAny<CancellationToken>()), Times.Once);
    }

    [Theory]
    [InlineData(0)]
    [InlineData(-1)]
    public void Constructor_WithInvalidAmount_ThrowsArgumentException(decimal amount)
    {
        Assert.Throws<ArgumentOutOfRangeException>(
            () => new TransactionRecord("id", amount));
    }
}