Felipe Moreira Ramos 1D AGO

“A good engineer knows when not to use AI.” I built a semi-automated workflow to summarize my live streams. With it, I can record and edit a stream in just one day of work. Not everyone has time to watch an entire stream, so creating a summary of the main moments is important for people who want to watch it later. Many tools create Shorts from videos by analyzing transcripts, but they are expensive and do not work well for visually heavy content such as gameplay. Gemini can analyze both video and audio and has a long context window, so I decided to use Google AI Studio to identify important moments. I also created scripts to automate parts of the video-editing process. Some of the scripts can be executed by an agent, and the prompts can be turned into Skills when using the API. Step-by-step: 1. I record the video with separate tracks for the microphone and background audio. I keep the microphone on the first track so the LLM does not identify only the background audio. 2. I compress and cut the video with `ffmpeg`, then extract the audio tracks. Google AI Studio has implicit video requirements: files must be under 400MB and under one hour long. The AI analyzes only one frame per second, so I can also lower the FPS to save space. I use the audio tracks later in the workflow. 3. I upload the processed videos to Google Drive, which makes them easier to use in Google AI Studio. 4. I use Gemini with temperature 1 and a high thinking level to select the important moments. I add the system prompt and specify which part of the video I am uploading: "The video is part X of the stream. Please make a structured script according to the system instructions." This helps Gemini understand what types of moments may appear in the video. The system prompt was created for gameplay presentations but can be adapted for other content. 5. I use Gemini Pro with temperature 1 and a high thinking level to find the timestamps for the selected moments. I keep timestamp generation separate from moment selection so Gemini has more thinking time for each task. Gemini Pro works better than Flash when handling time. I add the system prompt along with a copy of the response from the previous step. - As an extra check, after execution I continue the conversation with: "Check if the analysis was cut off too early (context truncation), ignoring that the video continued and generating false positives for timestamps. Check the last events especially." 6. I send the data to Blender. I create a JSON file containing the AI’s response and use a script to add the original video, the microphone track, and the background audio track. The script cuts and marks the important moments based on the JSON. Because it is not possible to send multiple audio tracks from a single video, I send the tracks separately. I also make sure to use the correct FPS, either 30 or 60. 7. I edit the video manually. The AI’s timestamps are not perfect, so I may add or remove sections, or correct a position that the AI identified incorrectly. I then render the video. 8. As an extra, I can automate standardized edits. For example, I use three different camera positions, so I add clips to three different tracks depending on the position I want. I use a script to change the position and scale of the clips on those tracks. 9. As another extra, I transcribe the audio. I mute the background audio and save only the microphone audio as an MP3, then use Whisper to transcribe it. I can use the transcript as YouTube subtitles or embed it directly into the video. I use Whisper-WebUI with Log probability Threshold -0.5, No Speech Threshold 0.5, Patience 2, and Hotwords `\u003cmy name and terms in other languages I usually use\u003e`. I use an LLM to translate the transcript into other languages. 10. Finally, I create tags and titles. I upload the transcript to Google Drive and use it in Google AI Studio with a system prompt to generate tags and titles for the video. I also use the assistant in Google AI Studio to analyze and summarize my channel for use in LLMs. This helps me choose better titles and tags.