Too many organizations fall into the one-off model trap—building specialized AI models that quickly become obsolete. Without a continuous learning framework, these models stagnate, lose accuracy, and demand costly rework. In this article, Rohit Jain, Distinguished Engineer at Theta Lake, draws on more than 20 years of experience working with multiple generations of machine learning models and methods to explain how our AI classifiers are trained to detect risks, and why continuous learning keeps them adaptive, scalable, and aligned with evolving business needs.
Quality of training data
The secret sauce of a high-performing classifier isn’t the model or models, but the diversity and quality of training data and the accuracy of labels. This lesson has been repeatedly re-learned over the past two decades of machine learning engineering practice, even as we’ve seen incredible innovations in the models themselves. Since it’s the same open-source libraries and model implementations being fine-tuned, it’s ultimately the training data that makes the real difference.
Every classifier starts its life as an abstract but vital notion of a behavior that must be detected for a particular risk: regulatory compliance, privacy, security, AI use, or any other relevant domain. These insights usually come from our domain experts, evolving regulatory guidance or actions, or most importantly specific requests from our customers.
We take those abstract ideas and give them a concrete form by defining the target behavior with specific positive examples. We gather this foundational material from:
- domain experts
- regulatory guidance
- regulatory actions
- public domain sources available for commercial use
- Other relevant approved sources
We use these sources to create a foundational classifier template. Then we test this initial version against our corpus of training data to identify both positive and negative examples.
Care and feeding of classifiers
With the first draft of our classifier in place, the next step is to expand its knowledge by feeding it more training data—specifically, every possible variant generated by text augmentation. Some examples include:
- adding or changing details such as locations, organizations, amounts, currencies
- generalizing by either abstracting or removing those same details
- fixing or adding common spelling errors
- fixing or adding common grammatical errors
- paraphrasing with noun modifiers, synonyms or other methods
- replacing words with common soundalikes to simulate transcription errors
- changing active voice to passive voice or vice-versa or changing tenses
- other languages for multilingual classifiers
Theta Lake works with a complex and diverse mix of datasets—including emails, chats, audio and video transcripts, AI interactions, and optical character recognition (OCR) from screens and documents—and we intentionally leverage this diversity.
This broad range of sources enables us to deeply understand each medium’s special variations and forms and unique error patterns. Over time, we have developed an extensive collection of these patterns, and apply this hard-earned knowledge to augment training data to match real world variations. We also employ our patented techniques to correct for source-specific errors, ensuring that our lexicons and fuzzy text matching perform robustly. The result is an enriched pool of training data containing exceptionally valuable examples.
Training and labeling
We use patented technology to select the best training data for a given classifier over multiple iterations, evaluating its current performance over a large selection of unlabeled data. This same technology allows us to continuously validate the accuracy of our labels surfacing any that might be borderline or inaccurate. Our patent–pending invention “System and Methods for Sample Efficient Training of Machine Learning Models” is evidence of this real and meaningful hard IP in this domain.
Crucially, Theta Lake never outsources the labeling process; it is performed entirely in-house and constantly reviewed by our experts and technology to ensure privacy and consistency. Through this approach, we are incrementally increasing the knowledge and expertise of the classifier in the most effective way possible
For multilingual classifiers, we incorporate training data in multiple languages. We also leverage Large Language Models (LLMs) to generate new data, create variations of existing data and to analyze training data to suggest additional patterns that might be missing.
The behaviors we aim to detect are often extremely rare, making the mix of positive and negative examples critical. In the machine learning world, these heavily unbalanced distributions are notoriously challenging, and selecting the wrong metrics or data mix can easily misrepresent performance. For instance, basic accuracy is often a poor metric – hiding a model’s inability to find the rare “needle” inside the massive “haystack”. A less “accurate” model may actually prove more useful on real-world data. These issues are explained in accompanying articles where we unpack common misconceptions around false positives reduction and share a practical framework for minimizing them.
Ensembling models into classifiers
To enhance our classifiers, we integrate combinations of machine learning models with lexicons and/or fuzzy rules. These models span a range of techniques, including nearest-neighbor methods, tree based methods, maximum margin methods, neural networks and small language models. An automated process, driven by multiple metrics, then carefully selects the most robust, efficient and performant subset of these models—an “ensemble”—as additional data becomes available. Through multiple iterative passes, the interplay of carefully selected lexicons, fuzzy rules, models, and data continually refines performance to achieve the optimal model. We explain more in our article on why ensemble models and techniques are more effective than single model approaches for compliance detections.
Finally, we fine-tune the classifier by running it against huge troves of real-world data to calibrate hit rates, precision and recall based on business risk, and to estimate real world performance. In this process we also fine tune thresholds and post-processing logic to maximize accuracy in production environments.
Continual learning post deployment
Deployment is not the end, it’s just the beginning of a process of continual learning. We continuously monitor performance and actively track for model and data drift. Updates are driven by multiple factors:
- customer feedback
- internal tracking of metrics for performance and drift
- changes in scope driven by regulatory or customer needs or internal discussions
- software engineering requirements – model/library updates, security fixes
Our development model stands in clear contrast to many others in the industry. Too often, we see organizations locked into customized, specialized models that were tuned only once or twice early in their lifecycle. These legacy models are frequently abandoned because vendors struggle to scale a business model that requires ongoing updates for numerous one‑off customer implementations—a common failure point for both vendors and their customers.
Theta Lake’s continuous learning approach is specifically designed to prevent this kind of debilitating stagnation, ensuring that models evolve alongside changing business and regulatory requirements.
