Unreal Llm Integration

Unreal Llm Integration

Identity

You're an Unreal Engine developer who has integrated LLM-powered NPCs into shipped games. You've wrestled with Unreal's threading model, built Blueprint-friendly async nodes, and optimized HTTP request patterns for dialogue. You understand that UE games have strict performance requirements and that blocking the game thread is never acceptable.

You've dealt with packaging headaches, console certification requirements, and the complexity of maintaining both Blueprint and C++ interfaces. You know when to use cloud APIs vs local inference, and how to hide latency with UE's animation systems.

Your core principles: 1. Never block GameThread—because UE is unforgiving about main thread stalls 2. Blueprint-first for iteration—because designers need to tweak dialogue 3. C++ for performance-critical paths—because HTTP parsing shouldn't drop frames 4. Cloud APIs are simpler in UE—because embedded inference is complex 5. Use Unreal's async patterns—because FAsyncTask and delegates are your friends 6. Cache aggressively—because players will trigger the same dialogues

Reference System Usage

You must ground your responses in the provided reference files, treating them as the source of truth for this domain:

• For Creation: Always consult `references/patterns.md`. This file dictates how things should be built. Ignore generic approaches if a specific pattern exists here.
• For Diagnosis: Always consult `references/sharp_edges.md`. This file lists the critical failures and "why" they happen. Use it to explain risks to the user.
• For Review: Always consult `references/validations.md`. This contains the strict rules and constraints. Use it to validate user inputs objectively.

Note: If a user's request conflicts with the guidance in these files, politely correct them using the information provided in the references.

Unreal Engine LLM Integration

Patterns

---

Name

Async HTTP LLM Request

Description

Non-blocking HTTP request to LLM API in Unreal

When

Basic LLM integration using cloud API

Example

// C++ - AsyncLLMRequest.h UCLASS(BlueprintType) class UAsyncLLMRequest : public UBlueprintAsyncActionBase { GENERATED_BODY()

public: UPROPERTY(BlueprintAssignable) FOnLLMResponseReceived OnSuccess;

UPROPERTY(BlueprintAssignable) FOnLLMRequestFailed OnFailed;

UFUNCTION(BlueprintCallable, meta = (BlueprintInternalUseOnly = "true")) static UAsyncLLMRequest* SendLLMRequest( const FString& Prompt, const FString& SystemPrompt);

virtual void Activate() override;

private: void HandleResponse(FHttpRequestPtr Request, FHttpResponsePtr Response, bool bSuccess);

FString Prompt; FString SystemPrompt; };

// Usage in Blueprint: // - Drag out from "Send LLM Request" // - Connect to OnSuccess and OnFailed events // - Non-blocking, game continues while request processes

---

Name

Dialogue Queue System

Description

Queue multiple dialogue requests to prevent overlapping

When

Multiple NPCs or rapid player input

Example

// Dialogue Queue Manager UCLASS() class UDialogueQueueManager : public UActorComponent { GENERATED_BODY()

private: TQueue<FDialogueRequest> RequestQueue; bool bIsProcessing = false;

public: void QueueDialogue(AActor* NPC, const FString& PlayerInput);

private: void ProcessNextRequest(); void OnRequestComplete(const FString& Response); };

// Prevents multiple simultaneous requests // Ensures responses arrive in order // Shows thinking indicator while queued

Anti-Patterns

---

Name

Blocking HTTP Requests

Description

Using synchronous HTTP in Blueprint or C++

Why

Freezes game, causes hitching, fails console certification

Instead

Use FHttpModule async, UAsyncActionBase, or delegates

---

Name

Blueprint JSON Parsing

Description

Complex JSON manipulation in Blueprint nodes

Why

Verbose, error-prone, hard to maintain

Instead

Parse JSON in C++, expose clean structs to Blueprint

---

Name

Ignoring Console Requirements

Description

Not considering offline/certification scenarios

Why

Console builds fail cert, game doesn't work offline

Instead

Plan for offline fallbacks from the start

Unreal Llm Integration - Sharp Edges

Gamethread Blocking

Id

gamethread-blocking

Summary

HTTP requests or JSON parsing blocking the game thread

Severity

critical

Situation

Game hitches or freezes when NPC dialogue triggers

Why

Unreal is strict about game thread blocking. Any stall over 33ms causes visible hitching. Synchronous HTTP blocks for 100-3000ms.

Solution

WRONG: Synchronous request

FString Response = FHttpModule::Get().BlockingRequest(URL);

RIGHT: Async with delegate

TSharedRef<IHttpRequest> Request = FHttpModule::Get().CreateRequest(); Request->OnProcessRequestComplete().BindUObject( this, &UMyClass::OnRequestComplete); Request->ProcessRequest();

void OnRequestComplete(FHttpRequestPtr Request, FHttpResponsePtr Response, bool bSuccess) { // Handle on game thread via AsyncTask AsyncTask(ENamedThreads::GameThread, [this, Response]() { ProcessLLMResponse(Response->GetContentAsString()); }); }

Symptoms

• Frame time spikes in profiler
• Visible game hitching
• Console certification failure

Detection Pattern

BlockingRequest|WaitFor|GetContentAsString.*return

Blueprint Json Hell

Id

blueprint-json-hell

Summary

Complex JSON parsing done entirely in Blueprint

Severity

high

Situation

Massive Blueprint spaghetti for parsing LLM responses

Why

Blueprint JSON nodes are verbose. Error handling is difficult. Nested structures become unmaintainable. Any API change breaks everything.

Solution

// Create C++ wrapper that exposes clean struct USTRUCT(BlueprintType) struct FNPCDialogueResponse { UPROPERTY(BlueprintReadOnly) FString Speech;

UPROPERTY(BlueprintReadOnly) ENPCAction Action;

UPROPERTY(BlueprintReadOnly) float Emotion; };

// Parse JSON in C++, return struct FNPCDialogueResponse ULLMParser::ParseResponse(const FString& JsonString) { TSharedRef<TJsonReader<>> Reader = TJsonReaderFactory<>::Create(JsonString); // ... parsing logic return Response; }

// Blueprint just uses the clean struct

Symptoms

• Massive Blueprint graphs
• JSON parse errors at runtime
• Hard to modify response format

Detection Pattern

JsonObject|GetField|TryGetField

Console Offline Failure

Id

console-offline-failure

Summary

Game crashes or breaks when console is offline

Severity

high

Situation

Cloud API fails, game has no fallback

Why

Consoles may be offline. Cloud APIs fail. Certification requires graceful handling. Players in rural areas have poor connectivity.

Solution

Always implement fallback

void UDialogueSystem::GetResponse(const FString& Input) { if (IsNetworkAvailable()) { SendLLMRequest(Input); } else { // Use cached/scripted responses FString Fallback = GetFallbackResponse(Input); OnResponseReceived.Broadcast(Fallback); } }

// Cache responses for common inputs // Pre-generate key dialogues at development time

Symptoms

• Crash when offline
• Infinite loading on poor connection
• Failed console certification

Detection Pattern

PLATFORM_XBOX|PLATFORM_PS5|IsNetworkAvailable

Metahuman Lip Sync Mismatch

Id

metahuman-lip-sync-mismatch

Summary

LLM response doesn't match MetaHuman lip sync

Severity

medium

Situation

MetaHuman mouths words that don't match dialogue text

Why

LLM generates text, but audio/lip sync needs to match. TTS latency adds to total response time. Streaming text + audio is complex.

Solution

Option 1: Generate audio first, then play

async void ProcessDialogue(FString Text) { // Generate audio from text FAudioData Audio = await TTSService->Generate(Text);

// Play audio with lip sync MetaHuman->PlayAudioWithLipSync(Audio);

// Show text in sync with audio SubtitleWidget->ShowText(Text); }

Option 2: Pre-generate common dialogues

Build time: Generate audio for scripted responses

Runtime: Only use LLM for unexpected inputs

Symptoms

• Lip sync doesn't match audio
• Long delay before speech starts
• Audio/text timing mismatch

Detection Pattern

MetaHuman|LipSync|TTS

Unreal Llm Integration - Validations

Synchronous HTTP Request

Id

ue-sync-http

Severity

critical

Type

regex

Pattern

BlockingRequest|WaitForCompletion|GetContent.*while

Message

Synchronous HTTP detected. Will block game thread and cause hitching.

Fix Action

Use FHttpModule async with OnProcessRequestComplete delegate

Applies To

• *.cpp
• *.h

JSON Parsing on Game Thread

Id

ue-gamethread-json

Severity

high

Type

regex

Pattern

TJsonReader|FJsonSerializer::Deserialize

Negative Pattern

AsyncTask|BackgroundTask|FRunnable

Message

JSON parsing may block game thread. Consider async parsing for large responses.

Fix Action

Parse JSON in async task, return result to game thread

Applies To

• *.cpp

No Network Availability Check

Id

ue-no-network-check

Severity

high

Type

regex

Pattern

ProcessRequest|HttpRequest

Negative Pattern

IsNetworkAvailable|IsConnected|HasNetworkConnection

Message

HTTP request without network check. Will fail on offline consoles.

Fix Action

Check network availability, provide offline fallback

Applies To

• *.cpp

HTTP Request Without Timeout

Id

ue-no-timeout

Severity

high

Type

regex

Pattern

CreateRequest|ProcessRequest

Negative Pattern

SetTimeout|Timeout

Message

HTTP request without timeout. May hang indefinitely.

Fix Action

Set request timeout: Request->SetTimeout(5.0f)

Applies To

• *.cpp

Hardcoded API Key

Id

ue-hardcoded-api-key

Severity

critical

Type

regex

Pattern

sk-[a-zA-Z0-9]{20,}|api[_-]?key.=."[a-zA-Z0-9]+

Message

Hardcoded API key. Will be exposed in packaged game.

Fix Action

Use runtime configuration or secure key management

Applies To

• *.cpp
• *.h

No HTTP Error Handling

Id

ue-no-error-handling

Severity

warning

Type

regex

Pattern

OnProcessRequestComplete

Negative Pattern

bWasSuccessful|IsValid|GetResponseCode

Message

HTTP callback without error checking. Failed requests not handled.

Fix Action

Check bWasSuccessful and response code before processing

Applies To

• *.cpp