From a Warm-Up Demo on November 3 to My Four-Phase AI Agent Roadmap

Most agents today run on static, hard-coded conditionals: if A, do B. The product I want to build has to answer a different question first — why does this matter to who I am right now?

Before I get to that ultimate goal, though, I have an assignment to finish. On November 3, I'm hosting a small sharing session with friends, and for that I plan to build a warm-up demo that turns light music into piano sheet music automatically. After that, I'll return to the real core product.

Lately my schedule has been packed with a job transition and various social commitments. But my thinking about AI, about how agents work, and about "who I am and how AI can genuinely empower an individual" hasn't stopped.

First, the assignment: turning light music into piano sheet music

The first thing I want to build is concrete: automatically convert a piece of light music into decent-quality piano staff notation, and perform it live through simulated playback.

This is not my ultimate product. It's a fun demo for the November 3 session, and at the same time a validation of the audio-processing and MIDI-generation pipeline. The technical path is roughly this: use an open-source audio-to-MIDI model (such as basic-pitch or MT3) to detect notes, run Python scripts for note quantization and noise trimming, render the result into PDF sheet music with music21 + MuseScore, and finally attach a SoundFont library to produce a high-fidelity simulated performance.

The goal is clear: at the session, show friends the complete flow from "input a piece of audio" to "one-click score plus a lively performance." For the audience it's a demo; for me it's also an early validation of the audio pipeline — get that pipeline working end to end in one concrete demo first.

The real goal: teaching an agent to know "why"

Once the warm-up is done, I'll go all-in on the core product.

Traditional agents mostly run on static, hard-coded conditionals. The other path I want to explore is using a Neo4j knowledge graph to give an agent a dynamically evolving "why" value. The graph should express "what kind of person I am" — my experiences, judgments, preferences, and the structure of my knowledge. Combined with what I say and the current context, the system reasons automatically and triggers the next action in concert.

In other words, the agent doesn't just execute instructions that have already been written. It first works out why something matters to who I am now, and then decides what to do.

The four-phase roadmap

Phase 1: Clear the immediate performance bottleneck

I've already got the Voice to Action glasses and engine working, but STT on my small home CPU machine is painfully slow. That's the most immediate pain point. Next I'll try the Groq API (Whisper large-v3), or faster-whisper (CTranslate2) locally, to see whether either can break through that performance ceiling.

This phase has another deliverable: completing the music-transcription warm-up PoC for November 3.

Phase 2: Put personal knowledge into Neo4j and watch how it collides with new expression

I write and accumulate knowledge every day. I'm curious: once that knowledge is turned into a Neo4j Knowledge Graph of entities and relationships, when the things I say and the new ideas I have collide with that graph, will it constrain what I say (Constraint) or expand it (Expansion)?

Going further, I want to bring in an Analogy model to observe where exactly that "constraint" and "expansion" happen after my personal body of knowledge meets new statements.

Phase 3: The Why → Action → Reaction chain

This is the most central step in the whole blueprint.

1. Why: combined with "my profile and context" in Neo4j, dynamically calculate why something matters to me now — that is, assign it a why value.
2. Action: trigger preset or dynamically generated Skills.
3. Reaction: feed the output of the action back into the graph, which automatically triggers the next action.

In this loop I'll introduce the Jev engine, so the knowledge graph can assign dynamic "why" values back to Skills, and get the automatic chain running. The system will try to "examine me" on its own, judging whether I need a new Skill Action — and even push me a hint before I've thought of it.

Phase 4: The ultimate form — Ambient IoT

Eventually it lands on a hardware product. It could be an Ambient Box on my desk, recording 24 hours a day, or an AI necklace/pendant I carry with me.

It shouldn't make me strongly aware that "I'm recording" or "I'm operating something." The ideal form absorbs information quietly, like air, and in the background judges, processes, and automatically generates animations or blog posts.

Two open questions, and one settled way of working

There are two questions on this path I still can't answer.

One is the outcome of the Phase 2 collision: will the knowledge graph make what I say more constrained, or open up more possibilities? I'll only know by actually building the graph and using it for a while.

The other is the Phase 4 hardware form. A desktop Box is closer to continuous capture in a fixed setting; a pendant is closer to something you carry and forget about. They point to different ways of using the product. I haven't decided, and I don't intend to lock it down before I've validated it.

What is settled is how I'll move forward. In August, I met three partners at a hackathon. We spent two months building an app together, and afterward found that we got along unexpectedly well. Our leader proposed that, since we clicked so well, we should each put our own goals on the table and push each other forward — and that focused conversation would help all of us. We ended up agreeing to check in roughly weekly. This blueprint is the thing I put on the table.

That agreement is a constraint on me: every week I have to show some real progress instead of leaving it all on the roadmap. The November 3 demo is the first checkpoint that can actually be tested.

Closing

From the small music-transcription demo on November 3 to building a dynamically evolving Why-Action-Reaction knowledge engine afterward, this is a long-term exploration of how AI can expand the boundaries of what one person can do.

If you're interested in music transcription, Neo4j knowledge graphs, or the dynamic decision mechanisms of AI agents, follow along. See you at the November 3 session.