video games

license: public domain CC0 

 

See further "spike" details here: spike with diagrams.

Hybrid AI‑Assisted Style Engine — Phase 0 Specification

A modular, Unreal‑integrated system for realtime per‑object neural stylization, style morphing, and expressive visual transformation.

---

TEXTUAL BLOCK DIAGRAM — FULL SYSTEM OVERVIEW

[ Offline Pipeline ]
    OASP → MMS → Style Variants + Metadata

[ Runtime Pipeline ]
    Level Loads → MSNS (One CNN per Level)
        ↓
    SMS (Style Morphing System)
        ↓
    RSTM (Runtime Style Transfer Module)
        ↓
    G-Buffer Strategy (Reuse / Multi-Res / Sequential)
        ↓
    Composite → Final Frame

[ Performance Layer ]
    BATS (Benchmark & Auto-Tuning System)

---

1. High‑Level Goals

• G1: Realtime per‑object neural stylization (hero feature)
• G2: Style morphing driven by gameplay context
• G3: Multi‑style CNN per level
• G4: Scalable G‑buffer & parallelization strategy
• G5: Benchmark & Auto‑Tuning System (BATS)
• G6: Optional offline AI asset upgrade pipeline (OASP/MMS)
• G7: Clean Unreal integration

---

2. Phase 0 Scope

• Unreal Engine only
• 1–3 stylized objects
• One multi‑style CNN per level
• Half‑resolution NST
• Octagonal zone demo room
• Basic BATS instrumentation

---

3. The “Sculpture Hall” Demo (Hero Experience)

[ Octagonal Sculpture Platform ]
    - One giant central statue (20 ft tall)
    - 6–12 smaller statues arranged around it
    - All statues have UStylizedObjectComponent

[ Surrounding 8 Zones ]
    - Flat-fronted alcoves arranged around the octagon
    - Each zone has a distinct theme:
        Victorian, Gothic, Sci-Fi, Underwater,
        Ink Sketch, Watercolor, Neon Glitch, Marble
    - Each zone has a rectangular trigger volume
    - Each zone has themed props + lighting

[ Player Movement ]
    - FPS camera
    - Entering a zone triggers a style morph
    - All statues morph toward that zone’s style

Why this demo works

• Extremely readable
• Visually dramatic
• Perfect for per‑object stylization
• Perfect for SMS morphing
• Perfect for sequential G‑buffer reuse
• Zero physics mismatch (statues are static)

---

4. Offline Asset Stylization Pipeline (OASP)

(Optional for Phase 0 — “but wait, there’s more”)

[ Base Mesh ]
    ↓
[ Multi-View Rendering ]
    ↓
[ Multi-View Diffusion ]
    ↓
[ Texture Reconstruction ]
    ↓
[ MMS Mesh Refinement ]
    ↓
[ Exported Style Variants ]

Outputs:

• Mesh variants
• PBR textures
• Style metadata

---

5. Meshing Model Subsystem (MMS)

(Optional for Phase 0)

Modules:

• Displacement refinement
• Procedural geometry augmentation
• Neural mesh refinement (Phase 1+)

---

6. Multi‑Style Neural Shader (MSNS)

One CNN per level, containing all styles for that level

[ Level Loads ]
    ↓
[ Multi-Style CNN ]
    ↓
[ Style Library (Embeddings) ]
    ↓
[ Per-Object Style Embedding ]

Unreal Implementation

• CNN implemented as a Global Shader (FGlobalShader)
• Weights stored in .uasset
• Style embeddings passed as constant buffers
• Compute dispatch via Render Graph

---

7. Style Morphing System (SMS)

Realtime interpolation between styles

[ Zone Style Embedding ] ----\
                               → [ Morph Interpolator ] → [ Current Style ]
[ Base Statue Style ] --------/

Driven by:

• Player entering a zone
• ZoneManager updating TargetStyleEmbedding
• Statues lerping toward target

Unreal Implementation

• AZoneManager tracks active zone
• AStyleZoneTrigger sets ZoneStyleId
• UStylizedObjectComponent lerps embeddings

---

8. Runtime Style Transfer Module (RSTM)

For each stylized object:
    Render into G-Buffer (Color, Normal, Depth)
        ↓
    Run CNN compute shader (MSNS)
        ↓
    Composite stylized output into scene color

Unreal Implementation

• Custom Render Graph pass
• FScreenPassTexture for G‑buffers
• RHICmdList.DispatchComputeShader for CNN
• Composite via fullscreen pixel shader

---

9. G‑Buffer Reuse & Parallelization Strategy

Parallelization Spectrum:
    [ Serialized ] — [ Hybrid ] — [ Fully Parallel ]

Resolution Pyramid:
    G0: 1/2 res
    G1: 1/4 res
    G2: 1/8 res
    G3: 1/16 res

Phase 0 Strategy

• Serialized (1 G‑buffer)
• Half‑res only
• Sequential per‑object processing

Unreal Implementation

• Allocate via GRenderTargetPool.FindFreeElement
• Reuse same RT for all objects
• Optional: add G1/G2 later

---

10. Editor Integration & Tooling

[ UStylizedObjectComponent ]
    - StyleId
    - StyleResolutionOverride
    - UpdateEveryNFrames
    - MorphSpeed

[ UStyleLevelSettings ]
    - StyleLibrary
    - CNNAsset
    - MaxParallelStylePasses
    - MaxStyleResolution
    - VRAMBudgetMB

[ ZoneManager + ZoneTriggers ]
    - ZoneStyleId
    - ZoneName
    - ZoneColorTint

---

11. Hardware Requirements

30 FPS → GTX 1060 / RX 580
60 FPS → RTX 2060 / 3060
120 FPS → RTX 3080 / 4070

---

12. Benchmark & Auto‑Tuning System (BATS)

[ Benchmark Mode ]
    ↓
[ Data Collection ]
    - CNN time
    - G-buffer time
    - Composite time
    - VRAM usage
    - Resolution tier usage
    - Temporal stability
    - Object count

[ Analysis Engine ]
    ↓
[ Auto-Tuning Report ]
    - Suggested resolution tier
    - Suggested parallelization
    - Suggested update frequency
    - Suggested object count

[ Apply Recommendations ]

Unreal Integration

• GPU timestamps (FGPUTiming)
• CPU trace events (TRACE_CPUPROFILER_EVENT_SCOPE)
• Unreal Insights for visualization
• Console command: StyleEngine.RunBenchmark 10

---

13. Unreal‑Specific Implementation Guidance

Global Shader Setup

• FStyleCNNShader : public FGlobalShader
• Bind:
  • InputColor
  • InputNormal
  • InputDepth
  • StyleEmbedding
  • CNN weights

Render Graph Integration

• Add pass after base lighting
• Use AddPass with compute dispatch
• Composite before post‑processing

Trigger Zones

• 8 UBoxComponent triggers
• Each with ZoneStyleId
• ZoneManager handles transitions

Statue Actors

• Static meshes
• No animation
• Perfect for stable G‑buffer inputs

---

14. Full System Flow

OFFLINE:
    Base Mesh → Diffusion → MMS → Style Variants

RUNTIME:
    Load Level → Load CNN → Load Style Library
        ↓
    Player Enters Zone → ZoneManager Sets TargetStyle
        ↓
    Statues Lerp Style Embeddings (SMS)
        ↓
    For Each Statue:
        Render to G-Buffer
        Run CNN
        Composite
        ↓
    Final Frame

BENCHMARK:
    Run BATS → Collect Stats → Generate Report → Apply Settings

---

15. Critical Risks & Mitigations

Physics Mismatch:
    → Use static statues only

Temporal Instability:
    → Use G-buffer inputs + optional temporal smoothing

VRAM Pressure:
    → Serialized G-buffer reuse

Style Interpolation Artifacts:
    → Train CNN with interpolation examples

Per-Object Overhead:
    → Limit to 1–3 statues in Phase 0

---

16. Applicability to SDF & Voxel Engines

SDF Engines:
    - MSNS: Full compatibility
    - SMS: Full compatibility
    - RSTM: High (needs masks)
    - OASP/MMS: Partial (needs SDF conversion)

Voxel Engines:
    - MSNS: Full compatibility
    - SMS: Full compatibility
    - RSTM: High (per-chunk stylization)
    - OASP/MMS: Limited (geometry dynamic)

---

17. Final Summary

This document defines a complete, Unreal‑focused Phase‑0 system for:

• Realtime per‑object neural stylization
• Style morphing driven by player movement
• Multi‑style CNN per level
• G‑buffer reuse & sequential processing
• Benchmarking & auto‑tuning
• A visually stunning “Sculpture Hall” demo

It is technically feasible, visually impressive, and architecturally clean.

---