hf_mirrors/various
Datasets
Dataset viewer
Files & versions

malcolmrey's Various AI Model, Architecture & Research Repository

Welcome to the central research and asset repository of malcolmrey. This repository hosts cutting-edge tools, custom architectures, RefMod latent adapter systems, video synthesis engines, training configurations, benchmark suites, cinematic scripts, and comprehensive educational guides spanning MiniMax-H3, FLUX.2 / Klein 9B, Qwen-Image 2.1, WAN 2.1, LTX-Video, Z-Image, SDXL, and Stable Diffusion.

Much of the training work here uses maltrainer, malcolmrey's own LoRA trainer.


🧭 Repository Map & Quick Navigation

Section / DirectoryFocus Area & DescriptionKey Resources & Direct Links
🎬 h3-center/MiniMax-H3 Video & Cinema Hub
Complete prompt guides, RefMod stacking, hybrid LoRA+RefMod containers, identity-LoRA epoch sweeps, 1,500+ character tests, and 9-episode Crossovers cinematic series.
• Prompting Guide
• RefMod Stacking Guide
• Hybrid LoRA+RefMod Guide
• Jinx LoRA Epoch Sweep & RefMod Stack
• Multistacking Benchmark Demos
• Known Characters Usability Index
• Crossovers Cinema Series
🎨 klein9/FLUX.2 & Klein 9B RefMods
Instant reference latent adapter ecosystem, custom ComfyUI nodes, CLI batch extractor, and benchmark suites.
• RefMod Architecture Guide
• ComfyUI Node
• Batch Extractor Tool
• Workflows | Visual Benchmarks
👁️ qwen/Qwen-Image 2.1 Multimodal Hub
Vision-Language (Qwen3-VL) conditioning architecture, reference strategy analysis, identity transfer, prompt engineering standards, edit samples, and a LoRA optimizer benchmark trained with maltrainer.
• RefMod Compatibility & ViT Architecture
• Reference Strategy & Prompting Guide
• LoRA Optimizer Benchmark (maltrainer)
• Qwen Hub Overview
🧪 moss/MOSS-Image LoRA Training
Combined training: one dataset trained with AdamW 8-bit, Automagic3 and Prodigy at short lengths, merged into one LoRA soup and resized to rank 8. Trained with maltrainer.
• Combined Optimizer Training Guide
• Member Templates (AdamW 8-bit, Automagic3, Prodigy)
⚡ zimage-turbo-vs-base-training/Z Image Base vs. Turbo Benchmark
Head-to-head training comparison across OneTrainer, MalTrainer, and AI Toolkit with datasets and weights.
• Benchmark Overview & Configs
• Datasets, AdamW/Prodigy configs, & .safetensors weights
🛠️ training-scripts/Production Training Configs
Ready-to-use recipes for OneTrainer, Musubi Tuner, AI Toolkit, and MalTrainer.
• OneTrainer Templates (Ernie, FK9, Krea2, SDXL, Z-Image)
• Musubi Pipelines (Ernie, LTX23)
• AI Toolkit (FK9, Ideogram 4, WAN, Z-Image)
• MalTrainer (Qwen-Image 2.1, Z-Image)
🎥 Sample Media & BenchmarksModel Generation Sample Libraries
High-fidelity outputs and verification test suites.
• ltx23-samples/ (20 LTX-Video 2.3 renders)
• ernie-samples/ (ernie-samples.zip)
• samples/fk9/ (Rose Byrne & Sydney Sweeney outputs)
• qwen/samples/edit/ (11 Qwen-Image 2.1 character edits)
🗂️ datasets/Sample Training Datasets
Small image sets used in the training experiments.
• datasets/jinx/ (19 images)
• datasets/karengillan/ (22 images)
📚 Guides & ArticlesComprehensive Knowledge Base
In-depth technical guides for model training, optimization, and prompt engineering.
• WAN 2.1 LoRA Training Tutorial
• CivitAI Articles Collection

🎬 MiniMax-H3 Center (h3-center/)

The MiniMax-H3 Center is an all-in-one research laboratory and production studio for the MiniMax Hailuo 01 / H3 text-to-video, image-to-video (I2V), reference-to-video (R2V), and clip-to-video (C2V) architectures.

📖 In-Depth Guides & Technical Documentation

  • 📘 MiniMax-H3 Video Prompt Writing Guide (v1.3.0): The authoritative standard for prompting MiniMax-H3. Covers pinpoint actor casting, character tag binding (<Subject 1>), timestamped action choreography (through 00:08.000), diegetic soundscapes, non-diegetic audio rules, camera vectors, and anti-patterns.
  • 🧬 RefMod Stacking & Multi-Subject Architecture Guide: Guide on injecting multiple pre-encoded latent references (.safetensors), single-persona Triple-RefMod fidelity stacking, conceptual body shape boosters, dual-persona shared frame conditioning, dual audio lip-syncing, and polyphonic counterpoint duets.
  • 📦 Hybrid LoRA + RefMod Container Specification Guide: Technical guide and implementation for embedding both LoRA weights and RefMod reference latents in a single .safetensors file, fully compatible with standard LoRA loaders (Power Lora Loader) and dedicated dual ComfyUI loader nodes.
  • 🎬 Multi-Stacking Benchmark Demos & Video Suite: Verified .mp4 benchmarks demonstrating single-persona Triple RefMods (Billie Eilish), dual conversations (Billie & Miley Cyrus), simultaneous two-shots, dual audio conditioning, and counterpoint singing duets with local RefMod latent files.
  • 🔗 C2V Continuous Chain Generation Guide: Pipeline for chaining 5-second to 15-second video clips losslessly into seamless continuous takes.
  • 🎵 Audio-Synchronized & C2V Music Videos Guide: Techniques for aligning video cadence and singing motion with pre-recorded vocal tracks.
  • ⚙️ RefMods Installation & Usage Guide & RefMod Creation Guide: Step-by-step instructions for extracting RefMod .safetensors from images and loading them into ComfyUI pipelines.
  • ⚖️ RefMods vs. Reference Images Comparison: Architectural analysis of speed, latent fidelity, and token efficiency comparing pre-computed RefMods against raw pixel reference workflows.
  • 🧪 Jinx Identity LoRA Epoch Sweep & RefMod Stack: A/B of an identity LoRA (epochs 60 / 80 / final) against a single identity RefMod and both combined, with 6 verified .mp4 clips. Findings: RefMod + LoRA is the strongest identity lock, epoch 60 is already enough, and the LoRA carries the Arcane art style along with the face.
  • 🐍 RefMod generator script: h3-center/docs/scripts/generate_refmod.py

👥 Known Characters & Usability Database

  • 📊 Usability Index (INDEX.md) & Unusable Index (INDEX_BAD.md): Empirical test results evaluating over 1,500+ characters, actors, and public figures directly in MiniMax-H3:
    • Good (540+ subjects): Characters with zero-shot likeness locks (e.g., House, Dexter, Michael Scott, Walter White, Gandalf, Wednesday Addams, Dean Winchester).
    • On the Fence (90+ subjects): Subjects requiring targeted prompt tuning or RefMod assistance.
    • Bad (920+ subjects): Subjects that fail without custom LoRAs or dedicated RefMods.

🎥 Crossovers Episodic Cinema Series

  • 🍿 Crossovers Series Master Catalog: 9 narrative episodes created with MiniMax-H3, with cast listings and synopses (8 rendered .mp4 files in the repo):
    • Episode 1: The Contagion of Secrets (House, Monk, Dwight, Dexter, Malcolm Reynolds, Mulder, Penny)
    • Episode 2: The Cosmic Extradition (Dean Winchester, Mal Reynolds, Sherlock, Pam Beesly, Lucifer, 10th Doctor, Saul Goodman, Steve Rogers, John Locke, Tony Stark)
    • Episode 3: Lockdown at Sabre Tower (Michael Scott, Wednesday Addams, John McClane, Seeley Booth, James Bond, Steve Rogers, Elliot Alderson, Lucifer, Jack Bauer)
    • Episode 4: The War for the Obsidian Portal (Maleficent, Daenerys, Gandalf, Dr. Strange, Hermione, Maximus, Jack Sparrow)
    • Episode 5: The Flat Earth Experience (Joe Rogan, Kanye West, Neil deGrasse Tyson, Dave Chappelle, Sir Anthony Hopkins)
    • Episode 6: Dave Chappelle: The Flat Earth Special (Stand-up comedy special)
    • Episode 7: Dr. Robert Ford: On Meaning (Philosophical monologue from Westworld)
    • Episode 8: Ellis Boyd 'Red' Redding: On Hope and Love (Shawshank monologue)
    • Episode 9: Captain Malcolm Reynolds: Alone on Serenity (Firefly character study)
  • 📚 Show & Character Guides (h3-center/crossovers/docs/shows/): Over 250+ show guides detailing character wardrobe, distinctive traits, and scene rules.

🎨 FLUX.2 & Klein 9B RefMods (klein9/)

RefMods (Reference Latent Adapters) provide zero-shot facial fidelity and concept locking for FLUX.2, Klein 4B, Klein 8B, and Klein 9B without the VRAM overhead or training time required by full fine-tunes or LoRAs.

  • 🌐 Interactive Model Browser: https://huggingface.co/spaces/malcolmrey/browser (Explore 1,400+ pre-extracted RefMods)
  • 🤗 Model Weights Hub: https://huggingface.co/malcolmrey/klein9 (Download .safetensors adapters)
  • 🔌 ComfyUI Custom Node: klein9/comfyui/custom_nodes/ComfyUI-Flux2Klein9Mod/ (Also available on GitHub)
  • ⚡ CLI Generator & Mass Batch Extractor: klein9/generate_flux2_klein9_refmod.py
  • 📑 Architecture & Extraction Guide: klein9/docs/FLUX2_KLEIN9_REFMODS_GUIDE.md
  • 📊 Visual Benchmarks & Comparisons: klein9/docs/VISUAL_BENCHMARKS_AND_COMPARISONS.md
  • 🔄 Production Workflows:
    • Pure RefMod Text-to-Image: klein9/workflows/workflow_klein9_refmod.json
    • Hybrid RefMod + LoRA Stack: klein9/workflows/workflow_klein9_refmod_lora.json
  • 🖼️ Sample Generations: samples/fk9/ (Rose Byrne and Sydney Sweeney benchmark runs)

👁️ Qwen-Image 2.1 Multimodal Hub (qwen/)

The Qwen-Image 2.1 Hub covers empirical research and architectural standards for image generation and character identity transfer using the Qwen-Image 2.1 Diffusion Transformer + Qwen3-VL Vision-Language conditioning pipeline.

📖 Technical Documentation & Guides

  • 📘 RefMod Compatibility & Vision-Language Architecture Analysis: Details why pre-encoded VAE RefMod adapters (.safetensors from MiniMax-H3 / FLUX.2) are mathematically incompatible with Qwen-Image 2.1, explaining Qwen3-VL's live ViT patch projection, dynamic token insertion, and joint cross-attention.
  • 📋 Reference Strategy, Prompting Standards & Identity Preservation Guide: Complete empirical benchmarking of reference strategies:
    • Demonstrates why 1 Reference Image per Person is the optimal standard.
    • Documents failure modes of multi-image referencing for single subjects (token dilution, feature blending).
    • Documents failure modes of 2x2 and 3x3 composite grids (grid leak, loss of high-frequency skin textures).
    • Provides master production prompt templates for precision face-swapping (preserving body, pose, age, skin pores, and clothing) and multi-character scenes.
  • 🏋️ LoRA Optimizer Benchmark: AdamW vs Prodigy vs Automagic3: Two-subject LoRA benchmark (Felicia Day, Rhea Seehorn) trained with maltrainer on one RTX 5090 directly from the ComfyUI INT8 checkpoints. Every checkpoint was scored for likeness (ArcFace), prompt adherence (Qwen3-VL judge), seed diversity and memorization.
    • Recommended: Automagic3, ~1500 steps (60–80 epochs), ≈13–16 minutes. Likeness plateaus by ~1250 steps and stays flat, with the best prompt adherence and the smallest adapter.
    • Likeness is a tie across optimizers; Prodigy's learning rate keeps climbing until it memorizes the training set and ignores the prompt.
    • Caption finding: if captions say only "a woman …", the trigger never learns gender, so prompt with "<trigger>, a woman, …".
    • Ready-to-use config: training-scripts/maltrainer/qwen_image_2_1_template.yaml

🖼️ Samples

  • qwen/samples/edit/: 11 Qwen-Image 2.1 reference-edit outputs placing actors into iconic roles (e.g. Al Pacino as Tony Montana, Charlize Theron as Furiosa, Mads Mikkelsen as Hannibal Lecter, Gillian Anderson as Dana Scully).

⚡ Z Image Base vs. Z Image Turbo Training Suite (zimage-turbo-vs-base-training/)

An empirical training study benchmarking Z Image (Base) against Z Image (Turbo) using identical curated datasets (Felicia Day, Olivia).

  • 📋 Overview & Documentation
  • 📁 Dataset: Curated 23-image high-resolution dataset
  • ⚙️ OneTrainer Configs: AdamW and Prodigy optimization profiles (.json)
  • 🤖 MalTrainer & AI Toolkit Recipes: Full training configurations (.yaml)
  • 📦 Trained Weights: Verified .safetensors models for both Base and Turbo versions

🛠️ Production Training Scripts & Templates (training-scripts/)

Curated configurations for high-efficiency model training across popular trainers:

  • 🟢 OneTrainer (training-scripts/onetrainer/):
    • fk9_template.json & fk9_prodigy_template.json — Klein 9B / FLUX.2 LoRA templates
    • ernie_template.json — Ernie training configuration
    • krea2_template.json & krea2prodigy_template.json — Krea 2 templates
    • sdxl_template.json & sdxl_prodigy_template.json — SDXL templates
    • zimage_base_template.json & zimage_turbo_template.json — Z-Image Base / Turbo templates
  • 🟣 Musubi Tuner (training-scripts/musubi/):
    • musubi/ernie/ — Multi-resolution Ernie training scripts
    • musubi/ltx23/ — LTX-Video 2.3 video model training configurations
  • 🔵 AI Toolkit (training-scripts/aitoolkit/):
    • fk9_template.yaml & fk9r_template.yaml — Klein 9B / FLUX.2
    • ideogram4_template.yaml — Ideogram 4
    • wan_template.yaml — WAN 2.1
    • zimage_template.yaml, zbase_template.yaml & zonetrainer_template.yaml — Z-Image variants
  • 🟠 MalTrainer (training-scripts/maltrainer/) — configs for maltrainer, malcolmrey's own trainer:
    • qwen_image_2_1_template.yaml — Qwen-Image 2.1 LoRA, the benchmark's recommended setting (Automagic3, ~1500 steps). Trains directly on the ComfyUI INT8 checkpoints. See the optimizer benchmark.
    • zimage_base_template.yaml — Z-Image Base LoRA

🎥 Video Samples & Benchmarks

  • 🎞️ ltx23-samples/: 20 full-motion test videos evaluating coherence, prompt fidelity, and motion dynamics with LTX-Video 2.3.
  • 📦 ernie-samples/: Comprehensive benchmark sample package (ernie-samples.zip).
  • 🎬 h3-center/crossovers/: 8 rendered .mp4 full narrative crossover episodes.
  • 🎼 h3-center/docs/multistacking/: 8 dual-audio and counterpoint singing benchmark videos.
  • 🧪 h3-center/docs/lora/: 6 Jinx identity-LoRA / RefMod comparison clips (5 s, 1344×768).
  • 🎤 h3-center/docs/examples/: 6 Billie Eilish / Miley Cyrus speech, singing and two-shot test clips.
  • 🗜️ ltx-artifacts.zip: 107 MB archive of LTX-Video material.

🗂️ Sample Datasets (datasets/)

Small image sets used in the training experiments (images only, no caption files):

  • datasets/jinx/ — 19 images (Jinx / Arcane)
  • datasets/karengillan/ — 22 images (Karen Gillan)

📚 Tutorials & Articles Collection

📘 WAN 2.1 LoRA Training Tutorial

Complete step-by-step guide on training WAN 2.1 LoRA models using AI Toolkit:

  • Dataset preparation (20 optimal images, 2500 steps)
  • Resolution bucketing & learning rate schedules
  • VRAM management & cloud GPU configurations
  • ComfyUI inference workflow integration

📚 CivitAI Articles Collection

Curated directory of malcolmrey's published training and generation guides:

  • DreamBooth / LyCORIS / LoRA Complete Guide
  • SDXL LoRA Training on RunPod
  • Textual Inversion / Embedding Training Guide
  • Flux Guide, Part I - LoRA Training
  • Improving Results with Multi-Model Blending (Turning it to 11!)
  • Quality Deep Dive (Bringing it up to Twelve!)
  • ADetailer, Inpainting & Facial Restoration Best Practices

🌐 Connect with malcolmrey

  • 💬 Discord Community: malcolmrey's place
  • ☕ Support / Priority Requests: Buy Me a Coffee
  • 🤗 Hugging Face Hub: https://huggingface.co/malcolmrey
  • 🔍 Interactive Model Browser: https://huggingface.co/spaces/malcolmrey/browser
  • 🎨 CivitAI Profile: https://civitai.com/user/malcolmrey
  • 💬 Reddit Community: r/malcolmrey