Feature Stores for Machine Learning at Scale

What Is a Feature Store?

A feature store is a centralised system for storing, serving, and reusing the input variables — features — that machine learning models are trained and scored on. It sits between raw data infrastructure and model training or inference, giving data science and engineering teams a single, consistent source of feature definitions instead of scattered scripts and spreadsheets.

The core problem a feature store solves is consistency between training and serving. A feature computed one way in a Jupyter notebook and a slightly different way in a production API is one of the most common — and hardest to debug — causes of model performance drift. A feature store enforces that the same feature logic and values are used in both places.

Why Do Growing Data Teams Hit a Wall Managing ML Features?

Data teams hit a wall when the number of models, features, and people touching them outgrows what a spreadsheet, a shared notebook, or a handful of cron jobs can coordinate. What worked for one data scientist and three models breaks down once five people are building features independently, with no shared registry of what exists or how it was computed.

The symptoms are familiar to anyone who has scaled a data function past its first few models:

• Duplicated logic. The same "customer lifetime value" or "days since last order" feature gets recomputed slightly differently by three different people, producing three different answers.
• Training-serving skew. A feature is computed in batch for training but needs to be computed in real time for serving, and the two implementations quietly diverge.
• No lineage. When a model's predictions degrade, nobody can quickly trace which upstream feature changed, when, or why.
• Stale documentation. A spreadsheet listing "our features" is out of date within weeks because nobody owns keeping it current.
• Slow onboarding. New data scientists spend their first few weeks reverse-engineering ad hoc pipelines instead of building models.

This is a scaling problem, not a tooling failure. The lightweight approach that got a team to its first production model is rarely the approach that supports its tenth. This is the same pattern we see across data infrastructure engagements more broadly — the pipelines and conventions that work for a small team become the bottleneck once the data function grows.

What Does a Feature Store Actually Provide?

A feature store typically provides a feature registry, an offline store for batch training data, an online store for low-latency serving, and a way to keep the two in sync. Some platforms extend this further into retrieval infrastructure that supports similarity or vector search directly against stored feature data.

Google Cloud's Vertex AI platform is a useful concrete example, and it's worth being precise about what its documentation actually shows rather than what the marketing implies. Vertex AI currently exposes feature management through two parallel resource architectures: a legacy featurestores resource, and a newer featureOnlineStores and featureGroups architecture aimed at feature management at larger scale. The newer architecture also supports nearest-entity search directly against feature data through a searchNearestEntities method, alongside Vertex AI's indexEndpoints — positioning the feature store as part of a broader retrieval layer that sits alongside RAG corpora and reasoning engines, not just a passive feature repository.

Feature stores on this platform also sit inside a wider MLOps resource surface — pipeline jobs, metadata stores with lineage tracking, hyperparameter tuning, model deployment monitoring, and Tensorboard integration. In other words, Google treats feature management as one component of a full-lifecycle ML environment, not a standalone product. Worth noting: this reflects the structure of Google's documentation and API surface, not an independent, vendor-neutral evaluation of feature store performance. Other established options in this space — both open-source and commercial — take different architectural approaches, and teams should evaluate against their own stack rather than any single vendor's framing.

Feature Store vs Lightweight In-House Tooling: How Do You Decide?

The right choice depends on how many models you run in production, how many people build features independently, and whether training-serving consistency has already caused a production incident. A dedicated feature store earns its complexity when the coordination cost of not having one exceeds the cost of adopting and operating the platform.

Signal	Lightweight in-house tooling	Dedicated feature store
Number of models in production	A handful, owned by one team	Many, owned by multiple teams
People writing feature logic	One or two people, tight coordination	Several people, independent workstreams
Training-serving consistency	Manually verified, low risk so far	Has caused or nearly caused a production incident
Real-time serving requirement	Batch scoring is sufficient	Low-latency online serving needed alongside batch training
Feature reuse across models	Rare — most features are model-specific	Common — the same customer or product features feed several models
Engineering capacity to operate new infrastructure	Limited; team is stretched	Available, or willing to use a managed offering
Governance and lineage requirements	Informal, low regulatory pressure	Formal — audit trail matters (common in fintech, healthtech, insurance)

For most growing Australian data teams, the honest answer at the early stage is: don't adopt a dedicated feature store yet. A well-documented set of shared transformation functions, a consistent naming convention, and a single source of truth for feature definitions — even if that's a version-controlled config file rather than a platform — will solve the majority of coordination problems for teams running a small number of models. Adopting a full feature store platform before you have the model count, team size, or serving requirements to justify it adds operational overhead without a corresponding return.

The calculus changes once you're running real-time inference across multiple products, have more than a couple of data scientists building features independently, or operate in a regulated sector — fintech, insurtech, healthtech — where auditability of exactly which feature values fed which prediction matters. At that point, the cost of building and maintaining bespoke consistency guarantees in-house tends to exceed the cost of adopting a managed or open-source feature store.

How Should a Data Team Evaluate a Feature Store Platform?

Evaluate a feature store the same way you'd evaluate any core piece of data infrastructure: against your actual serving latency requirements, your existing cloud and data warehouse footprint, your team's operational capacity, and your governance obligations — not against a vendor's feature list in isolation. A platform that's technically capable but poorly integrated with your existing pipelines will add friction rather than remove it.

Questions worth asking before committing:

• Does this integrate cleanly with the cloud and data warehouse we already run on, or does it require a parallel data platform?
• Do we need real-time online serving, or is batch scoring sufficient for our use cases today?
• Who on the team will own operating this, and do we have the capacity?
• What does migration look like if we outgrow this platform, or if the vendor changes its architecture — as Google's shift from a single featurestores resource to a split online/offline architecture illustrates can happen even with major providers?
• Does the platform give us the lineage and audit trail our regulatory obligations require?

These are architecture decisions, and they're easiest to get right early — retrofitting feature governance onto a sprawling set of ad hoc pipelines is considerably harder than designing for it from the start. This is the kind of decision we help clients work through as part of broader AI engineering and AI product strategy engagements, where the question is rarely "which platform" in isolation but how feature infrastructure fits the rest of the data and ML stack.

Where Horizon Labs Fits

We help Australian data and engineering teams decide whether a dedicated feature store is worth adopting, and if so, which architecture fits their existing cloud footprint and regulatory obligations. That usually starts with an honest audit of current feature pipelines and model count, not a platform recommendation on day one. We've written more on the foundational data work this depends on in our insights, including how to think about the broader data infrastructure that feature stores sit on top of.

If you're weighing up feature store adoption against continuing with lightweight in-house tooling, we can help — starting with a clear-eyed look at where your team actually sits on that curve before recommending any platform.