Keyvector | Dev notes

One key per row: turning a Cassandra primary key into a vector key

Mon, 02 Nov 2026 00:00:00 GMT

Every vector in S3 Vectors has a key, a string of up to 1,024 characters. Keyvector has to choose one for each row, and three different paths have to agree on it:

the stream, when a row changes;
the backfill, when it reads existing rows through CQL;
the query, which turns a hit back into a row to read.

Use the whole primary key, written so it can be read back

AWS’s sample used a single text column as the key. That breaks the moment a table has a composite primary key, which in Cassandra is most tables. Keyvector uses all the primary key columns, sorted by name, written as column=value pairs joined by &, with each part percent-encoded:

movie_id=12&studio=Redwood%3DRidge

Sorting makes the key independent of column order. Percent-encoding means a value containing & or = can’t be mistaken for a separator. Our test table deliberately has studios called “Northlight & Co” and “Redwood=Ridge”. And because the key is reversible, a query hit can be turned straight into WHERE movie_id = ? AND studio = ? without storing anything else.

The harder part: the same value has to look the same twice

A row reaches Keyvector in two shapes:

from the change stream, every scalar is a string: an int is "42", a timestamp is text in a format AWS’s docs don’t pin down;
from CQL, values are typed Java objects: Integer, Instant, UUID, ByteBuffer.

If the two produce different text for the same value, the backfill and a later update write the same row under two keys, and the index silently holds both.

So both paths normalize to one form before a key is built:

every integer type becomes a Long;
float is read through its shortest decimal form, so 0.1f is 0.1 and not 0.10000000149;
a timestamp becomes epoch milliseconds, whether it arrived as digits, as ISO-8601 or as CQL’s yyyy-MM-dd HH:mm:ss.SSSZ;
a blob becomes hex.

A unit test feeds every key type through both paths and requires identical keys. The live test then backfilled 302 rows and streamed 61 changes over them, ending with exactly one vector per row.

Keys that are too long

S3 Vectors keys stop at 1,024 characters. A row whose encoded primary key is longer can never be stored, and retrying won’t change that, so Keyvector logs it and skips it rather than stalling the shard behind it forever. In practice a primary key that long is rare.

What broke when we ran it against real AWS

Mon, 02 Nov 2026 00:00:00 GMT

Unit tests can tell you a planner makes the right decision. They can’t tell you what Bedrock does to a new account, or what KCL does the first time it starts. So we built a test kit (e2e/) and ran Keyvector against a real Keyspaces table, a real change stream and real S3 Vectors indexes. We embedded with both Amazon Bedrock Titan V2 and a local Hugging Face TEI running all-MiniLM-L6-v2. The whole thing cost well under a dollar.

The sync logic held up: every one of 61 live changes was accounted for, a restart resumed at exactly the right place, and stopping the model server mid-run lost nothing. Here’s what didn’t hold up.

Titan V2 allows 60 requests a minute, and you can’t raise it

On a new account, on-demand Titan Text Embeddings V2 is limited to 60 requests a minute, and Service Quotas lists that quota as not adjustable. Titan embeds one text per request, so that’s one row per second. A 1M-row backfill through Titan would take about 11.5 days.

That turned up our first bug: the Bedrock embedder didn’t retry throttling. It decided what to retry by asking the SDK’s retryable() and searching the error message for “ThrottlingException”. But the SDK reports a throttle it has already retried as not retryable, and the message says “Too many requests”. So the first throttle failed the whole batch. In the stream, the processor’s own batch retry would eventually have saved it; in a backfill, the run simply died, though it had written nothing.

Two fixes:

Retry on the actual signal: a throttling or 429 status, a 5xx, or a network error.
Pace requests to the quota in the first place. Keyvector now spaces requests across all its threads to a per-minute limit, which defaults to 60 for Titan. The rerun of the same backfill took 5½ minutes for 320 rows, with zero throttles.

The bigger lesson is about the product, not the code. If a table changes faster than one row a second, or needs a big backfill, use TEI, Cohere Embed (96 texts per request) or Bedrock batch inference. “Bring your own model” turns out to be about throughput, not only privacy.

The first start waits 12 minutes

On its first start, the stream worker did nothing for twelve minutes. KCL 3 adds a secondary index to its DynamoDB lease table and assigns no shard until the index is active. That took about 12 minutes per new application name, both times we measured. Later starts are immediate. It’s documented now, because “nothing is happening” is the worst thing a first-time user can see.

Shutdown could hang forever

While KCL was waiting on that index, stopping the worker hung. Our shutdown hook waited for KCL’s graceful shutdown without a timeout, and KCL can’t finish shutting down while it’s still starting, so only kill -9 worked. Now the hook gives up after 30 seconds. Nothing is lost by giving up: an unacknowledged batch is simply read again by the next worker.

A busy port killed the startup thread

Restarting quickly put a second worker on the same health-check port. Binding failed, the exception escaped main, and the worker kept running with no health endpoint and a scary stack trace. Now it logs the problem and carries on.

`aws login` credentials didn’t work

The fork’s AWS SDK for Java (2.33) predated aws login, the CLI’s browser sign-in, so the connector couldn’t use the credentials every new AWS user now has. Upgrading to 2.46 and adding the signin module fixed it. The Python side needed boto3[crt] for the same reason.

The other surprises

A new account’s Bedrock access is blocked while AWS verifies the account, for up to two hours. keyvector check caught it before anything ran, which is the reason it exists.
AWS’s “new AWS experience” accounts can’t use Amazon Keyspaces at all unless you activate advanced features. We tested in a classic account.
KCL logs Incomplete hash range found every two minutes against Keyspaces streams. It expects Kinesis-style hash ranges, which these streams don’t have. It’s noise; processing is unaffected.

Why we forked the AWS sample, and what we changed

Mon, 02 Nov 2026 00:00:00 GMT

In April 2026 AWS published a blog post and a sample repository showing how to stream Amazon Keyspaces changes into an S3 Vectors index. It’s MIT-0 licensed, it reads the change stream with the Kinesis Client Library (KCL), and it handles the genuinely hard parts of stream processing: leases, checkpoints and shard splits. We kept all of that.

The vector connector itself is a demo, though, and reading it as if it would run in production turned up five problems. None of them raises an error. Each one silently leaves the index wrong.

1. A failed batch was skipped, silently

When applying a batch failed, the processor logged the error and returned. KCL then delivered the next batch, and when that one succeeded, checkpointed past both. The rows in the failed batch never got a vector, and nothing anywhere said so.

Keyvector retries a failing batch with backoff and doesn’t save the worker’s position until the batch is applied. After a configurable number of attempts it parks the rows’ keys in a dead-letter queue and moves on. Without a queue it keeps retrying: a stalled shard is a visible failure (the stream’s age climbs), a skipped batch isn’t.

2. Deletes were ignored

The sample read only each change’s new image. A deleted or expired row has none, so its vector stayed in the index forever, and searches kept finding rows that no longer existed.

3. Every environment variable was printed at startup

The runner logged its whole environment, so an embedding API key would have ended up in CloudWatch Logs. Keyvector logs no environment and reads keys from a named variable that ECS fills from Secrets Manager.

4. The deployment template was too broad, and wrong

The CloudFormation template gave the task’s execution role full ECS access, and its task role dynamodb:* and cassandra:* on every resource. It also set the table-name variable to the keyspace name.

Keyvector’s template scopes every permission to one table, its stream, the connector’s own lease tables, one vector index and one queue (see IAM permissions).

5. Two settings were quietly ignored

The stream: with several streams on a table (CDC switched off and on again), the sample took whichever was listed first, which could be a disabled one that never delivers anything. Keyvector takes the newest enabled stream.
The region: the sample configured the Cassandra driver’s SigV4 plugin with a key named region. The plugin reads aws-region, so the setting did nothing and only worked because AWS_REGION happened to be set.

One trap that isn’t in the sample

A stream can record new images, old images, both, or keys only, and the choice can’t be changed later. A connector that reads “no new image” as “row deleted”, which is the natural way to detect deletes, would read every insert and update on a KEYS_ONLY or OLD_IMAGE stream as a delete and empty the index. Keyvector checks the stream’s view type at startup and refuses to run on one without new images.

The sample did its job, which was to show the pattern. Keyvector’s job is to keep an index right for months without anyone watching.

Keyvector | Dev notes

One key per row: turning a Cassandra primary key into a vector key

Use the whole primary key, written so it can be read back

The harder part: the same value has to look the same twice

Keys that are too long

What broke when we ran it against real AWS

Titan V2 allows 60 requests a minute, and you can’t raise it

The first start waits 12 minutes

Shutdown could hang forever

A busy port killed the startup thread

aws login credentials didn’t work

The other surprises

Why we forked the AWS sample, and what we changed

1. A failed batch was skipped, silently

2. Deletes were ignored

3. Every environment variable was printed at startup

4. The deployment template was too broad, and wrong

5. Two settings were quietly ignored

One trap that isn’t in the sample

`aws login` credentials didn’t work