Your prototype works. One Qdrant node, a thin API, a vector search that returns in 20 ms. Then a customer asks for a bigger cluster on Friday afternoon. Another is mid-re-index and slowing its neighbour. A third wants their data deleted, along with proof of deletion.

None of those is a search problem. They are the other half of the product, the half that was never in the demo, and it is where most of your engineering will go.

This post walks through that half using one imagined service, a managed vector store built on Qdrant and run for about 2,000 customer clusters. One customer will go through the whole thing.

A customer of that service assumes you do all of the following, and the demo shows none of it.

Each line is a small system. Together they are larger than the engine. The rest of the post sorts them into two piles and shows why one pile deserves most of your attention.

The data plane answers the customer's requests. The control plane sets things up, changes them, watches them and fixes them.

For this service the sort is short.

The test for any feature is one question. Does it run for every customer request, or does it change what is running? Searching a collection is data plane. Creating a cluster to hold that collection is control plane.

That sort points to a design goal: answering a customer's request should not depend on the control plane being up. If the control plane goes down, you can't create or resize anything, but existing clusters should keep answering searches.

There are exceptions, and it is better to name them. Checking the API key and limiting request speed often happen on every request. The answer is not to forbid that. Let those checks read from a local copy that the control plane keeps filled. Then a control plane outage means the checks use slightly old settings, and requests still go through.

When the customer clicks create, that is not one action. It is a series of steps across separate systems, and they can't be rolled back together as one unit.

Any step can fail, and your own worker can crash between any two steps. Say it crashes after step 3 and the customer clicks create again. If your code starts over from the top, it reserves three more nodes, then fails because the DNS name already exists. Now you are paying for nodes nobody uses, and the customer sees an error.

The fix is to stop treating create as one API call and treat it as a resumable workflow, a job that can stop at any point and carry on later. First, write down what the customer asked for (a cluster row in state PENDING, with the size and name). Then a worker goes through the steps and records each one as it finishes. If the worker crashes, the next worker reads the row and carries on from the first step that is not done.

This is what keeps the promise from the top of the post. A crash or a failed step means the worker picks up where it stopped, so the customer gets one cluster, not two. If the cluster never becomes healthy, the worker undoes the steps it already did and marks the cluster FAILED, which the customer sees as a clear error. Billing only starts in the last step, so a cluster that never came up never costs them anything.

Two things make this work. Every step must be idempotent, meaning running it twice gives the same result as running it once. Running "create DNS name" twice must leave one name, not an error and not two. And the row must record what is done, not only what is wanted, or the worker can't tell step 3 from step 4.

This pattern has a name from Kubernetes: the reconcile loop. A worker keeps comparing what the customer wants with what actually exists, and does the next thing that closes the gap, over and over. The Kubernetes docs on controllers explain the idea well. You do not need Kubernetes to use it. You need a table of what is wanted, a table of what exists, and a worker that is fine running the same step twice.

Resize and delete work the same way, and are harder. If a resize fails halfway, the old size must still be serving, so you add the new capacity before you remove the old. Delete is the one where the customer wants proof, so it ends by writing a record that says the volumes were wiped, the keys destroyed and the backups set to expire. That record is a product feature.

Now the reverse. Say a bad config change leaves 40 clusters unable to start. This is exactly when the control plane must work, because it is what restarts them, undoes the config change and tells you which 40 are broken.

That is why health checks, support access and repair tools must not depend on the thing they repair. If the health dashboard reads from the same Qdrant nodes it reports on, it goes blind during the outage it exists for. Health data should come from small agents and probes that send their reports out to a store somewhere else.

Three things follow from this.

The control plane needs its own uptime target too. Keep the part it can't do without small and simple, so there is less to break.

Billing looks like a counter. It is really a pipeline, and the numbers have to be right in a way the data plane never needed.

For that customer, the service might charge for node-hours, stored gigabytes and number of requests. Each one has to be measured on the nodes, sent to the control plane, added up per customer per hour, and multiplied by a price. Along the way, usage records arrive late, arrive twice, or never arrive because a node was replaced mid-hour.

You will see "exactly once" in descriptions of this. What you can really achieve is effectively once. Collect each record at least once, give each a unique ID, and drop duplicates when adding up. For late records, close each hour only after a short wait, and let later corrections change a past invoice line in the open, not silently.

The bar is that an invoice can be checked. A finance person, or a doubtful customer, picks one line and traces it back to the usage records. If you can't, a small rounding bug becomes a trust problem. That is why metering is its own system with its own tests, not a column on the cluster table.

Now the noisy neighbour. On a Tuesday, that customer starts a re-index and sends 5,000 vectors per second instead of 200. On shared nodes, the customer on the same machine sees their slowest searches (the p99, the slowest 1%) take three times as long. A noisy neighbour is one customer's load slowing down another's, and the blast radius is how many customers one failure or spike can reach.

You have three tools, and you normally use all of them.

Dedicated is not always the answer. It costs more per customer, and you have more clusters to upgrade and watch. Most managed services end up offering both, and move the customer who keeps hitting limits to dedicated. Before you promise anything, check how your engine keeps customers apart.

The service needs to move 2,000 clusters from Qdrant version N to N+1. Nobody wants a maintenance window, and customers in different time zones all believe their night is the quiet one.

The common approach is a rolling upgrade with canaries. You try the new version on a small group first and widen it only if that group stays healthy. Upgrade a few of your own clusters first, then 1% of customers, then 10%, then the rest, with a health check between each stage. If errors or slowness in the upgraded group go past a set limit, the rollout stops and goes back. Inside one cluster, nodes upgrade one at a time, and the next node is only touched once the upgraded one has caught up.

Be honest about what you promise. "No planned downtime in the common case" is achievable. "Zero downtime for every change" is not, because a change to how data is stored on disk, or a cluster with a single node, may need a window. Tell customers about those cases in advance.

A backup you have never restored is only a hope. Who is responsible for not losing the data? In a managed service, you are, and the customer assumes it is handled.

The practice is a scheduled restore test. Take a snapshot of a sample of clusters, restore each into a spare cluster, and compare point counts and a few known queries against the original. Record how long it took, because that is your real recovery time. Do it all the time, not once.

Tenancy means every request, and every action by your staff, is tied to a specific customer. In practice this is four decisions. How do customers log in (API keys limited to one cluster and one role)? How does traffic reach the cluster (a public endpoint with an allow-list, or a private network)? How is data encrypted at rest (one shared key, a key per customer, or a key the customer holds)? And who inside your company can reach which cluster?

Each option has a cost. A key per customer limits the damage of one leak, but adds a key-management service you now run. A private network closes off one kind of exposure, but adds setup steps for the customer. This is not a security guide. Treat it as a list of questions to bring to a security review, not as answers.

One more job decides whether the business makes money, and that is planning capacity and cost. The control plane decides which node hosts which cluster (placement), when to add machines, and when to shrink a cluster that has been idle for a week. Those choices set your cost of goods, what it costs you to run what you sold. A cluster sitting at 5% CPU on a dedicated node costs the same as a busy one. Whether a managed service makes money is decided in the control plane, not in the engine.

So what do you build first? If you have the engine working and a blank page, this order is a sensible default.

The rule behind the order is simple: build first what you can't add later (the state record, repeatable steps, usage records), then what hurts customers most.