[Dev Log][Python] Create short videos from photos and clips with Gemini 3.7 Flash: ReelCraft

I noticed a new page in the Gemini API documentation called Omni, introducing a model named Gemini Omni Flash, described as "natively multimodal, processing text, images, audio, and video simultaneously." My first thought was straightforward: if I throw a whole folder of videos and photos from my phone into it, let it understand what each asset is about, and then tell it in one sentence to edit them into a short video—isn't that a video editing app?

After reading the documentation, I realized I had misunderstood, and the misunderstanding happened to be at the most critical point. However, after bypassing that limitation, the rest was actually feasible. The result is ReelCraft: a Python CLI where you feed in a bunch of videos and photos, Gemini 3.7 Flash understands the assets one by one and provides editing suggestions. Once I confirm the edit list, ffmpeg cuts it into a 9:16 vertical short video, background music is generated using Lyria 3, and subtitles are automatically burned in.

Along the way, there were three issues where both ffmpeg and Gemini reported success, but the output was wrong—the kind of errors you only discover by actually playing the video.

This article will cover: Omni Flash is not what I thought it was Bypassing limitations: Per-file understanding, then text aggregation Using edl.yaml as a human confirmation point The difference after switching to Gemini 3.7 Flash Background music: Lyria 3 uses a different API ffmpeg will silently fail your edits Subtitles: Two issues only visible after burning them in Other pitfalls Conclusion Reference links

Gemini Omni Flash (gemini-omni-flash-preview) is a video generation and editing model that uses the Interactions API. It allows you to use natural language to apply effects to a single video, such as "when the person touches the mirror, make the mirror ripple beautifully like liquid." It is not a tool for "understanding a bunch of videos."

Referencing or reasoning across multiple videos is not supported. Attempting multi-video prompting may result in degraded model performance or unexpected outputs.

Video references up to 3 seconds in duration are accepted by the API schema but are not correctly processed by the model at this time.

So the path of "throwing a bunch of videos in and letting it understand and edit them" was blocked for Omni Flash. The models that can actually perform multi-video understanding are the standard Gemini models: starting from version 2.5, a single request can include up to 10 videos. With a 1M context window, it can handle about an hour of footage at default resolution, tokenize it second-by-second, and output scene descriptions with timestamps.

The time spent on this misunderstanding wasn't wasted. The verification process helped clarify "which task should be handled by which model," and the architecture followed naturally.

The key design decision is in the second and third steps: call Gemini once for each video to get precise internal timestamps and descriptions; then feed these text results (not the raw videos) into a second call for cross-asset aggregation, sequencing, and editing suggestions.

This approach has two benefits. First, it completely avoids the "multi-video reasoning not supported" issue because the second call only sees text, not ten videos. Second, it isn't limited by the 10 videos/request cap; no matter how many assets there are, it just means more independent calls in the analyze phase. Those calls can be retried or fail individually without affecting each other.

Testing also proved that timestamps are more reliable when processed separately. When asking about ten videos in a single prompt ("which seconds are the highlights?"), the model easily confuses the timelines of different videos.

Failure handling in the analyze phase is recorded separately: if a file fails after three retries, it's logged in analysis/errors.json, while other files continue. This later revealed a loophole during review, which I'll discuss later.

I decided from the start not to make it "one-click fully automatic." Between inputting assets and outputting the final product, there must be a place where I can manually intervene, because LLM-provided edit points will inevitably have some irrationalities, and re-running the entire pipeline incurs API costs again.

Videos use in/out to mark the range, photos use durationsec for duration, and note is the reason for selection written by Gemini (this field was later used for subtitles, see below). To change an edit point, just change the numbers; to change the order, move the clip; after saving, run poc render.

The outputs of each stage remain in the project directory, so any step can be re-run individually. analyze also skips files that already have analysis results, so re-running doesn't incur double charges—this is very helpful when iterating on prompts.

poc plan --theme was added later: you can provide a sentence as the editing theme, e.g., --theme "Participating in the COSCUP open source community". This affects the narrative angle of the summary, the priority of clip selection, and the wording of each clip's note. Since it only affects the plan stage, changing the theme doesn't require re-analyzing assets, making it very cheap to try different narratives on the same set of materials.

The understanding and aggregation stages initially used gemini-2.5-flash, then switched to gemini-3.7-flash. This is the GA stable version, not a preview: