Qwen3.8で動画解析→生成AI動画プロンプト化してみた。LM Studioも対応

ffmpeg -y -i test/test.mp4 -vf "scale=-2:480" -c:v libx264 -crf 28 test_480p.mp4

curl -X POST "http://192.168.11.100:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"Qwen/Qwen3.8-Flash-Next\",
\"messages\": [{
\"role\": \"user\",
\"content\": [
{\"type\": \"text\", \"text\": \"この動画の内容を解析してください\"},
{\"type\": \"video_url\", \"video_url\": {\"url\": \"data:video/mp4;base64,$(base64 -i test_480p.mp4 | tr -d '\n')\"}}
]
}]
}"

{"id":"chatcmpl-a97f30d928ad852e","object":"chat.completion","created":1789660693,"model":"Qwen/Qwen3.8-Flash-Next","choices":[{"index":0,"message":{"role":"assistant","content":"この動画の内容は、若い女性がカフェでスイーツを食べる様子です。彼女は白い服を着ており、まず店に入ります。その後、店員と会話をして、ケーキとアイスカフェオレを頼んでいるようです。最後に、彼女はケーキを一口食べて、美味しいという表情を浮かべています。","refusal":null,"annotations":null,"audio":null,"function_call":null,"reasoning":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null,"routed_experts":null}],"service_tier":null,"system_fingerprint":"vllm-0.29.0-ed12bfe0","usage":{"prompt_tokens":5995,"total_tokens":6059,"completion_tokens":64,"prompt_tokens_details":null,"completion_tokens_details":{"reasoning_tokens":0}},"prompt_logprobs":null,"prompt_token_ids":null,"prompt_text":null,"kv_transfer_params":null,"ec_transfer_params":null,"metrics":null}

pip install mlx-whisper

import mlx_whisper
import json

result = mlx_whisper.transcribe(
    "test/test.mp4",
    path_or_hf_repo="mlx-community/whisper-large-v3-turbo",
)
print(json.dumps(result, ensure_ascii=False, indent=2))

{
"text": "今日は六本木をブラブラしてるんだけどいいお店ないかな?わ、ここ!入ってみよう!いらっしゃいませーすすみません、いちごのケーキ一つくださいはい、かしこまりましたうん!おいしい!ここ、やばい、店員さんイケメン",
"segments": [
{
"id": 0, "seek": 0, "start": 0.0, "end": 2.82,
"text": "今日は六本木をブラブラしてるんだけどいいお店ないかな?",
"tokens": [50365, 43791, 30566, 8802, 41437, 5998, 28889, 11353, 28889, 11353, 8822, 4895, 26983, 21891, 13806, 6117, 22469, 9311, 24939, 30, 50506],
"temperature": 0.0, "avg_logprob": -0.2417571518447373,
"compression_ratio": 1.5257731958762886, "no_speech_prob": 1.775446629992405e-11
},
.
. (長くなるので省略)
.
{
"id": 7, "seek": 0, "start": 12.06, "end": 14.64,
"text": "ここ、やばい、店員さんイケメン",
"tokens": [50968, 23509, 1231, 7355, 42176, 1231, 22469, 20992, 15567, 8040, 32639, 21647, 4824, 51097],
"temperature": 0.0, "avg_logprob": -0.2417571518447373,
"compression_ratio": 1.5257731958762886, "no_speech_prob": 1.775446629992405e-11
}
],
"language": "ja"
}

pip install panns-inference

import wave, numpy as np
from panns_inference import AudioTagging
from panns_inference.config import labels

with wave.open("test_audio_32k.wav", "rb") as w:
    data = np.frombuffer(w.readframes(w.getnframes()), dtype=np.int16).astype(np.float32) / 32768.0
audio = data[None, :]  # (1, samples)

model = AudioTagging(checkpoint_path=".panns/Cnn14_mAP=0.431.pth", device="cpu")
clipwise_output, _ = model.inference(audio)

probs = clipwise_output[0]
order = np.argsort(probs)[::-1][:10]
for i in order:
    print(f"{probs[i]:.3f}  {labels[i]}")

0.851  Speech
0.312  Inside, small room
0.224  Babbling
0.156  Child speech, kid speaking
0.093  Giggle
0.062  Baby cry, infant cry
0.050  Whimper
0.047  Baby laughter
0.037  Belly laugh
0.030  Crying, sobbing

integrated_multimodal_description:

[Shot 1] Live-action, vlog style, handheld selfie medium close-up. A young woman with long brown hair, wearing a white lace top, cream cardigan, and plaid skirt, walks down a sunny street in Roppongi. The camera shakes slightly as she moves forward, capturing storefronts and pedestrians in the background. She looks around casually and speaks to the camera with a clear, casual voice (S1): <d>[Japanese] 今日は六本木をブラブラしてるんだけどいいお店ないかな?</d>

[Shot 2] At 00:03.50, the camera pans right as the woman spots a cafe entrance. Her expression brightens with excitement. She turns and walks through the glass door. The young male staff member inside (S2) greets her politely: <d>[Japanese] いらっしゃいませーす</d>

[Shot 3] At 00:06.00, the shot cuts to a static handheld view inside the cafe. The woman stands at the counter facing two young male staff members in white shirts. She orders: <d>[Japanese] すみません、いちごのケーキ一つください</d> The staff member (S2) responds: <d>[Japanese] はい、かしこまりました</d>

[Shot 4] At 00:09.00, the shot cuts to a static medium shot of the woman sitting at a wooden table. A strawberry shortcake and an iced drink are in front of her. She takes a bite with a spoon, chews, and nods happily, saying: <d>[Japanese] うん!</d> She takes another bite and exclaims with delight: <d>[Japanese] おいしい!</d>

[Shot 5] At 00:12.00, the shot cuts back to a handheld selfie view on the street. The woman is walking again, smiling broadly and talking to the camera with a playful tone: <d>[Japanese] ここ、やばい、店員さんイケメン</d>

overall_soundscape: Ambient city noise including distant traffic and footsteps on pavement. The sound of a door opening and closing, followed by the quiet hum of a cafe interior with coffee machines and soft chatter. The clink of a spoon against a plate and the rustle of clothing as the subject moves.

non_diegetic_music: N/A

{
"style": "live-action, vlog, handheld selfie, urban, casual",
"shots": [
{ "index": 1, "time_range": "0.0s-3.0s",
"subjects": "Young woman with reddish-brown hair, wearing a white lace camisole, white knit cardigan, and a pastel plaid skirt. She is holding the camera with her right hand.",
"actions": "The woman walks down a city sidewalk while talking to the camera. She looks around, scanning the shops on the left.",
"environment": "City street in Roppongi, daytime, overcast lighting. Shops and trees line the street." },
{ "index": 3, "time_range": "5.0s-8.0s",
"subjects": "The woman (back/side profile) at the counter. Two male baristas in white shirts and black aprons standing behind the counter.",
"actions": "The woman stands at the counter talking to the staff. The baristas smile and listen.",
"environment": "Cafe interior, counter with coffee machines and pastries, warm lighting, white tiled wall." }
],
"speakers": [
{ "id": "S1", "description": "Young female, on-screen, bright and energetic tone, fast speaking rate, Japanese accent" },
{ "id": "S2", "description": "Male, on-screen (barista), polite and calm tone, standard speaking rate, Japanese accent" }
],
"dialogue": [
{ "time": "0.0s",  "speaker": "S1", "text": "今日は六本木をブラブラしてるんだけどいいお店ないかな?", "confidence": "high" },
{ "time": "3.16s", "speaker": "S1", "text": "わ、ここ!入ってみよう!", "confidence": "high" },
{ "time": "4.94s", "speaker": "S2", "text": "いらっしゃいませーす", "confidence": "high" },
{ "time": "6.0s",  "speaker": "S1", "text": "すみません、いちごのケーキ一つください", "confidence": "high" },
{ "time": "8.08s", "speaker": "S2", "text": "はい、かしこまりました", "confidence": "high" },
{ "time": "9.0s",  "speaker": "S1", "text": "うん!", "confidence": "high" },
{ "time": "11.16s","speaker": "S1", "text": "おいしい!", "confidence": "high" },
{ "time": "12.06s","speaker": "S1", "text": "ここ、やばい、店員さんイケメン", "confidence": "high" }
]
}