Github: https://github.com/perfact-tang/AIDirector_AI-81builders

From a Story to a Video: Three Layers I Validated at a Hackathon

At this hackathon, I built a video generation tool. When I finished, what made me happier than “building a tool” was that it validated three things for me: whether video can be produced continuously, whether AI services can be packaged as products that ordinary people can use, and whether sound can become the starting point for video generation. All three layers worked.

Layer One: Turn One-Off Generation into a Continuous Pipeline

The mainstream approach to AI video generation today is to use a first frame, a last frame, and a prompt. For a single clip, that is enough. But when you actually make a video, the problem shows up immediately: after finishing one segment, it is hard to connect the next one. The further you go, the more painful the transitions become, and the workload keeps growing.

My approach changes the starting point: the user only passes in a story or a blog post, and the tool automatically decides how many sections the story needs, then generates the prompt for each section. It also generates characters and voiceover, and produces the video section by section. The project is complete.

The value here is not “it can generate video.” It is that “start from zero every time” becomes “break it down first, then produce section by section.” It is the same story, the characters and voices can carry over, and each section only has to solve the visuals for its own part. For me, that is the key to whether video production can be sustained.

Layer Two: Wrap the CLI in a Service Layer So Non-Programmers Can Use AI

This layer comes from my instinct as a software engineer. First, I want to thank AI& for providing a substantial credit sponsorship, which let me really put these AI services to work with confidence this time.

I have been to many hackathons, especially in the past year, mostly “vibe coding” hackathons. People mainly build products with AI tools, and those products are basically aimed at programmers: call various AI APIs, then package the capabilities as AI services for programmers. Those products make sense, but the user boundary is clear—only people who can write code and call APIs can use them.

I kept thinking: if AI& can run these AI services to their fullest, can they serve not only programmers but also ordinary people?

Hearing AI& introduce their product this time, what they offer now is the AI& CLI service. The CLI is a familiar entry point for developers, but it can also be a foundation: wrap the CLI into a service layer, give that service layer a frontend, and non-programmers can use these capabilities. If we then add our own features on the server side, it is no longer just a command-line tool—it becomes an intelligent service.

Based on this idea, I did two things:

1. Installed the skills I wrote into this service layer.
2. Connected the service layer to the AI& CLI tool: the frontend passes in customer requirements, and the service layer uses my skills plus the AI& engine to call the AI LLM API, execute and drive those skills, split the script and generate it piece by piece, and finally return a standard JSON.

The idea worked. The frontend does not need to understand the internal structure of a skill, and the service layer does not need to rewrite a workflow for every requirement. The two sides connect through a standard JSON.

Following the same line of thought, I noticed that AI& had launched a new feature: calling MiniMax’s video model. Reading the docs, I saw that this video model also supports sound-guided video generation. So I built a feature in the service layer that I call Action: after the frontend sends information, the service layer calls the AI& CLI in the background, then calls MiniMax’s model to generate video. That also got done.

What surprised me was the price: video generation costs only 10 yen per second. I generated a video of about one minute, and it cost 500–600 yen in total. That was far below my expectations—and remarkably generous.

Looking back, AI& did the job excellently, and I turned it into a server-side command. More importantly, this architecture can be reused widely: build a good frontend, make AI the core entry layer on the server, and configure Action and Skills to provide services. That lets AI&’s capabilities reach more uses, and lets developers deliver better, smarter services to ordinary users.

Layer Three: Sound First—Let the Voiceover Decide the Visuals

The third layer is a new approach I discovered in this practice and want to keep advocating: using sound to drive video generation.

The problem was concrete: videos generated section by section often lack continuity. My solution was not to pile on more prompts, but to write the voiceover first. The test is simple: with no visuals, just listening to the voiceover, can you feel the story? If you can, half the job is done.

Next, cut the voiceover into segments, and use each segment’s voice as a reference for MiniMax to generate the corresponding video. The model refers to the meaning in the voice to imagine actions and generate content. Compared with writing a large amount of visual description in prompts, this approach is more practical: the voice itself carries emotion, rhythm, and intent, and the generated video has a core intent and purpose driving it. The visual result was also very good.

Three Validations, All Successful

At this hackathon, I completed three validations, and all of them succeeded: a video system that can produce continuously, a service-layer architecture that brings AI capabilities to ordinary users, and a sound-first method of video generation. For me, that is worth more than simply building a demo.

Finally, thanks to the organizers, +81 Builders, for bringing excellent sponsors, a great venue, and very tasty pizza. I hope the event keeps getting better.