Send meeting audio via Telegram to my local AI agent for transcription, speaker identification, summaries, and action items
I record a meeting on my phone and send the audio file to my Telegram chat with Hermes, my local AI agent. That is the only trigger—there is no app, upload page, or extra step. Hermes runs on my PC through WSL and handles the rest locally, except for the LLM analysis. 1. **Send the audio via Telegram.** I drop the meeting recording into my Telegram chat with Hermes. 2. **Create the meeting record.** When Hermes receives the audio message, it calls a FastAPI endpoint at `localhost:8200`. The endpoint creates a meeting record and uploads the file. It is a thin Python server backed by SQLite, using async SQLAlchemy. 3. **Transcribe the meeting locally.** The backend runs faster-whisper on my PC, so there is no cloud transcription API or transcription cost. It handles large files and produces the full text with timestamped segments. 4. **Identify the speakers.** A diarization pass determines who spoke when. I can optionally upload a 10-second voice sample for each person; the system stores MFCC features and can automatically label speakers such as “Steve” and “Andrew” in future meetings. Diarization is best-effort, so transcription still works if it fails. 5. **Analyze the transcript with an LLM.** The transcript and timestamped segments are sent to DeepSeek through the OpenRouter API. A single structured prompt extracts: - A TL;DR - Thematic topic groups - Decisions made - Open questions - Risks flagged - A participant list - Tags - Proposed action items, each with an owner, priority, due date, confidence score, source timestamp, and the exact transcript quote it came from 6. **Send action items to the todo app.** Each proposed task has a **Create in Todo** button. With one click, I send it to my separate local `agent-native-todo` service on port `8100`, along with the meeting title, evidence quote, and timestamp as context. The todo app runs independently; the meeting app only calls its REST API. 7. **Review everything in the web miniapp.** The results appear in a React SPA at `/miniapp/`, which includes a meeting list, FTS5 full-text search, transcripts with clickable timestamps that jump to the corresponding point in the audio, color-coded tasks, speaker name editing, and audio playback with seek. The stack is: - **Backend:** Python FastAPI, async SQLAlchemy, and SQLite - **Transcription:** faster-whisper, installed with `pip install faster-whisper` - **Diarization:** pyannote.audio or a similar tool; optional because transcription works without it - **LLM:** OpenRouter API with DeepSeek, using one structured prompt and JSON mode - **Frontend:** React, Vite, and plain CSS without a framework - **Agent glue:** Hermes Agent receives Telegram messages and orchestrates the API calls - **Todo integration:** Any task app with a REST API; the meeting app only needs to POST to it The only part that costs money is the LLM analysis step, which costs about $0.01 per meeting through DeepSeek. Everything else runs on my PC.
0 comments