Drive the LTX Director node

LTX Director (Timeline)

LTXDirector (pack: WhatDreamsCost-ComfyUI, category WhatDreamsCost) is a video timeline editor node. Its on-canvas UI has Add Image, Add Text and Add Audio buttons.

You cannot click those buttons. Don't try.

They are not node inputs. The pack builds them as raw DOM elements with JS handlers:

addTextBtn.addEventListener("click", () => this.addTextSegmentFreeSpace());

Panel/MCP tools drive the LiteGraph node model (widgets + inputs). They cannot invoke arbitrary DOM handlers, so "click Add Text" is impossible. This is a hard limitation, not flakiness. Say so plainly rather than retrying.

The real control surface: timeline_data

Everything those buttons do is serialized into ONE string widget, timeline_data. The pack's own code treats it as "the absolute source of truth". Set it and the editor renders it.

It is listed in the pack's HIDDEN_WIDGET_NAMES, so it does not appear in the node's visible widget list, but it is an ordinary input and is settable:

create_workflow(action="modify", workflow=<the graph>, operations=[
  { op: "set_input", node_id: "42",
    input_name: "timeline_data", value: "<json string>" },
])

The value is a JSON string, not an object. Empty state is "{}".

Sibling hidden widgets: local_prompts, segment_lengths, guide_strength, audio_data, use_custom_audio, inpaint_audio, use_custom_motion, override_audio.

Schema (verified against a real production workflow)

{
  // TRACK GATES — a track's segments are IGNORED unless its gate is true.
  "mainTrackEnabled": true,
  "audioTrackEnabled": false,
  "motionTrackEnabled": false,

  // must mirror the node's start_frame / duration_frames widgets (see Edges)
  "normalStartFrame": 0,
  "normalDurationFrames": 4393,

  "global_prompt": "",
  "overrideAudio": false,
  "inpaint_audio": false,

  // retake mode
  "retakeMode": false, "retakeStart": 24, "retakeLength": 48,
  "retakePrompt": "", "retakeStrength": 1, "retakeVideo": null,
  "retake_global_prompt": "",

  // display-only
  "propHeight": 90, "globalPropHeight": 60, "showFilenames": true,

  "segments": [],        // main track: image AND text segments
  "motionSegments": [],
  "audioSegments": []
}

Text segment (segments)

{ "id": "seg1", "start": 0, "length": 48, "prompt": "wide shot, neon city", "type": "text" }

Image segment (segments)

{
  "id": "1784657553220x4fxu",
  "start": 0,
  "length": 241.68,
  "prompt": "",
  "type": "image",
  "imageFile": "whatdreamscost/Nintendo_ZeldaUrbosaHotspringKF_00002_.png",
  "imageB64": "/api/view?filename=Nintendo_ZeldaUrbosaHotspringKF_00002_.png&type=input&subfolder=whatdreamscost"
}

imageB64 is a misnomer. It holds a /api/view URL, not base64. So an image segment points at a file already in ComfyUI's input dir. Full agent-drivable recipe:

1. upload_image (action:"image") → puts the file in the input dir (note its subfolder/name)
2. imageFile = "<subfolder>/<name>.png"
3. imageB64 = "/api/view?filename=<name>.png&type=input&subfolder=<subfolder>"

Audio segment (audioSegments)

{
  "id": "17846562234967gjvk", "type": "audio",
  "start": 0, "length": 4393, "trimStart": 0, "audioDurationFrames": 4393,
  "audioFile": "whatdreamscost/Intergalactic Hip Hop.mp3",
  "fileName": "Intergalactic Hip Hop.mp3",
  "waveformPeaks": [0.047, 0.274, 0.537, "…~200 floats…"]
}

waveformPeaks is the rendered waveform. It is cosmetic. Omit it or pass [] if you're writing a segment programmatically; the audio still plays. Don't fabricate plausible-looking peaks and imply they were measured.

Edges (get these wrong and it silently does nothing)

1. Track gates. Pushing into audioSegments does nothing while audioTrackEnabled is false. Set the gate in the same edit. (The reference workflow ships an audio segment with the track OFF, which is easy to misread as broken.)
2. Frame bookkeeping must agree. normalDurationFrames == duration_frames widget == segment_lengths widget (observed: all 4393), and normalStartFrame == start_frame. Change the timeline length and you must update all of them.
3. Frames, not seconds. start / length are pixel-space frames and may be fractional (241.68). duration_seconds / frame_rate / time_units are display concerns.
4. global_prompt exists twice, inside timeline_data and as a forceInput socket on the node. In the reference workflow the socket is wired from a PrimitiveStringMultiline; prefer the wired source and keep them consistent.
5. guide_data must go somewhere. LTXDirector.guide_data → LTXDirectorGuide. That partner node is what injects the keyframes.
6. Unsupported by the pack. The widget tooltip says "auto-managed; do not edit by hand". Well-formed blobs load fine; malformed JSON will break the editor. Round-trip and re-read after writing, and tell the user this is unofficial.

Reference wiring (from a working LTX 2.3 Director graph)

Power Lora Loader (rgthree) ─ model ─┐
DualCLIPLoader ──────────── clip ────┤
VAELoaderKJ ───────────── audio_vae ─┤   LTXDirector #42
PrimitiveStringMultiline ─ global_prompt ─┘
        │ model         → LTX2_NAG
        │ positive      → ConditioningZeroOut, LTXVConditioning
        │ audio_latent  → LTXVConcatAVLatent
        │ guide_data    → LTXDirectorGuide   ← required partner
        │ frame_rate    → AudioToFrameCount, VHS_VideoCombine
        └ combined_audio→ AudioToFrameCount, PreviewAudio, …

Procedure

1. Find the node: get_workflow (action:"query") with types: ["LTXDirector"], fields: "detail" (or panel_query_graph on the live canvas).
2. Read the current timeline_data and JSON.parse it. Never hand-splice the string.
3. Mutate the parsed object (append a segment, flip a track gate, retime).
4. Keep normalStartFrame / normalDurationFrames in sync with the widgets (edge 2).
5. Write it back as a string via set_input.
6. Re-read to confirm it round-trips, then create_workflow (action:"validate").

Related

PromptRelayEncodeTimeline (pack: ComfyUI-PromptRelay) carries the same timeline_data hidden-widget pattern, so this skill's approach applies there too.

Any node whose controls are DOM widgets over hidden state is invisible to clicking and must be driven through its state widget. LTX Director is the clearest example.

Sources

• Official: none found.
• Empirical: timeline_data schema and edge rules from a working LTX 2.3 Director graph.