An hour of video in a single vector

The system

A personal AI assistant. Users connected sources, the platform ingested them into a vector index, and users then asked questions across everything they had connected — or had the platform generate content from it.

The sources were accepted at a finer grain than “a link”:

Source	Subtypes
Twitter	tweet, thread, retweet, profile, hashtag, cashtag
YouTube	single video, entire channel
Medium	article, account
News	direct URL, or by keyword
PDF	by URL, or uploaded
Financial	crypto and stock insights
Generic	any URL

Uploads additionally covered PDF, JSON, CSV, TXT and DOCX. Output fanned out past text into image generation, text to speech, speech to text, and video.

Everything went in whole

Here is the thing I would change first if I built it again.

There was no chunking. A tweet and an hour-long YouTube transcript were handled identically: extract the text into a single field, embed it, store one vector. Langchain was a dependency, but no text splitter was ever used. The chunk_size and chunk_overlap fields that exist in the codebase belong to a third-party webhook type that these sources never touched.

For a tweet, that is correct and cheap — forty words is already about the size of a chunk.

For an hour of transcript it is close to useless. One embedding of a ten-thousand-word document is an average of everything the document is about, which means it is a precise representation of nothing. A user asking a narrow question — one specific claim made at minute 43 — is matching their question against the blurred centroid of the whole video. There is no passage to retrieve, because no passage was ever stored separately.

Which is why retrieval was wrong, and why nobody knew

The roadmap carried a line item to improve metadata vectors: a public admission that retrieval was not good enough.

The failure had the shape you would predict from the paragraph above. Short, dense sources — tweets — outrank long, useful ones, because a tweet’s single embedding actually is about its forty words, while an article’s single embedding is about everything at once. The system confidently returned the tweet.

And it was never measured. No evaluation set, no benchmark questions, no retrieval scoring. The problem was visible enough to reach a roadmap and it shipped anyway, while the team moved to the next milestone.

That is the ordinary condition of most AI features that ship. The retrieval layer is assumed to work until somebody complains, and “somebody complains” is not a metric.

What I would build first now

Not chunking. A test set — representative questions with the correct source labelled for each one.

That ordering is deliberate, and it is the thing I actually learned here. You cannot tell whether chunking, a re-ranker, or metadata filtering fixed anything without ground truth to score against. Each of those is a guess at the fix. The test set is what tells you whether the guess helped, did nothing, or made things worse, and by how much. Ship the fix first and you are optimising blind: no baseline, no way to catch a regression, no way to compare one approach with another.

The test set also tells you what to build second, because the failures cluster into patterns once you can see them.

Sources that keep producing

Several source types are not documents but subscriptions — a profile, a channel, a hashtag, a news keyword all keep emitting new content after a user adds them once.

Those ran on Agenda, a MongoDB-backed scheduler, re-polling once a day.

Deduplication was at the application layer rather than a database constraint: before inserting, the job looked for an existing entity matching source type, subtype and value, and skipped it if found. When a user added a source, the platform also checked whether that value already existed globally and reused the existing entity instead of scraping it again — so two users following the same channel did not produce two copies of it.

An application-level check is not a uniqueness guarantee. Two jobs racing on the same value can both read “not found” and both insert. At the volumes involved it never mattered. A unique index is what actually enforces it.

Generation ran inside the request

Image generation takes tens of seconds. Text to speech and video take longer.

All of it ran synchronously inside the HTTP request — the server called the provider and polled it in a loop within the same request/response cycle, then returned the result. No job queue. No websocket or server-sent events. The browser simply held the connection open and waited.

That works until it doesn’t. A long request is exposed to every timeout between the browser and the provider — proxies, load balancers, the browser’s own limits — and there is no way to recover a job whose connection dropped, because nothing recorded that the job existed.

Nothing retried

There was no retry or backoff for external providers anywhere: no rate-limit handling, no exponential backoff. Scheduled jobs that threw were caught by a generic failure handler that logged the failure to a history collection and stopped there.

Logging a failure is not handling it. For a pipeline whose whole job is calling other people’s rate-limited APIs on a daily timer, retry with backoff is the first thing I would add after the test set.

Identity and publishing were separate

Worth stating precisely, because it is easy to describe wrongly.

Sign-in was Auth0. The OAuth connections to Twitter and Medium were not authentication — they were for publishing generated content out to those platforms, with tokens stored in their own configuration collection.

Adding Twitter as a knowledge source went through an entirely different endpoint that never consulted those tokens. Connecting your Twitter account for posting did not add it as a source, and adding it as a source did not connect your account. Two systems that look like one from the outside.

Result

No numbers. No beta user counts, no volume of sources indexed, nothing I can point at — and there is a reason they are not recoverable.

The product no longer exists in this form. Knowlee has since pivoted to an enterprise system, and what it sells today has nothing to do with the personal assistant described above. The consumer platform, its seven source types and its content generator were not iterated on; they were replaced.

That is an ordinary startup outcome and it says nothing about the engineering. It does mean the honest answer to “how did it do?” is that I do not know, and nobody is going to tell me.

What I can say is that this was a team of three and I was the tech lead — which is also why the faults above are described the way they are. Reviewing your own architecture honestly a year later is more useful than defending it.