Establishing an AI-native foundation is critical for effective compliance oversight, the ongoing challenge lies in keeping those machine learning models accurate, scalable, and reliable over time. Building on the multimodal framework detailed in Part 1, this article explores how the rigorous disciplines of continuous monitoring, smart data curation, global scaling, and the pragmatic integration of Large Language Models (LLMs) turns massive communication data streams into actionable compliance intelligence.
Measuring and Monitoring: Living Models
With extensive domain expertise across our data science team, we knew from day one that rigorous metrics and continuous monitoring are vital for keeping classifiers healthy. We have all seen machine learning companies build hyper-customized models for each individual customer site—a strategy that results in unmaintainable codebases that slowly drift out of date and are eventually abandoned. In contrast, Theta Lake’s core models are standardized across all customers, while risk thresholds and minor operational parameters can be safely fine-tuned. This architecture allows us to monitor, maintain, and update our models consistently.
We update our models dynamically based on customer feedback regarding performance in terms of false positives or negatives, as well as our internal monitoring of hit rates and concept or model drift. We also deploy updates immediately in response to security or library updates– tracking all performance metrics via central internal dashboards.
Early on, we realized that data-driven development would require specialized regression analysis tooling. We invested heavily in building a proprietary framework to run these analyses across text and image data. Over time, we added advanced capabilities for cloud-based sampling, automated execution, and deep metrics collection. This framework is invaluable for testing ensembles of different algorithms in a principled manner, ensuring every single classifier update performs as well as, or better than, its predecessor.
Needles in Haystacks: The Right Metrics
An inherent challenge of the compliance domain is the high rate of false positives. Risky, non-compliant, or unprofessional behavior typically accounts for less than 0.1% of total corporate communication flows—which makes training machine learning classifiers incredibly difficult.
Any classifier designed to ensure it catches all bad behavior will necessarily err on the side of caution, even if it’s borderline, occasionally flagging benign behavior as suspicious. The goal is to minimize those false positives while maintaining the true-positive rate: which is known as the needle-in-a-haystack problem.
One of the most important decisions to make this work is the right choice of metrics to optimize against and monitor. There is a lot of research out there on this and still no best known method that applies across all datasets. We’ve done extensive experimentation to pick and choose the right metrics for our classifiers.
To solve this, optimizing and monitoring against the correct metrics is critical. Despite extensive academic research, there is no single universally accepted methodology that applies to all datasets. Through years of extensive experimentation, we have engineered a proprietary matrix of metrics specifically optimized for our classifiers.
Data: The Secret Sauce
Data is the lifeblood of our models. Our proprietary “Smart Labeling” technology U.S. Patent No. 12,664,235 allows us to intelligently curate training data from vast datasets, minimizing errors while maximizing data diversity and efficiency. The system also runs automated checks on labels and surfaces potentially mis-labeled data for human experts to verify.
This ensures that during each training iteration, our classifiers actively sample under-represented or unrepresented data, allowing us to develop classifiers much more efficiently in terms of dataset size and compute time.
Going Global
While our initial classifier development focused heavily on English, we designed our platform to scale into a multilingual world. We began expanding our capabilities into common European languages, eventually adding complex CJK (Chinese, Japanese, Korean) languages.
The primary hurdle in multilingual compliance is accurate and robust language identification, which is exceptionally difficult when analyzing short snippets of text within an email or a chat room. This challenge is further complicated by “code-switching”—the fluid switching between multiple languages within a single conversation or record. To solve this, we engineered custom in-house language detection test suites.
Through detailed error analysis of real-world performance degradation, we discovered that we also needed to detect noisy audio and explicitly quantify transcription reliability as a core confidence metric for our classifiers. In some cases, detecting silent intervals within noisy audio streams proved vital for flagging and preventing incorrect classifications. The fine-tuned smaller language models—trained on our intelligently selected and accurately labeled datasets alongside highly targeted multilingual examples—allow us to seamlessly generalize our core English models across global languages.
The Generative Shift: LLMs and Beyond
As Large Language Models (LLMs) matured, we integrated them selectively and pragmatically. As early access beta partners with vendors like Anthropic, we evaluated foundational models well before their public release. We built and deployed our first LLM-powered feature—chat summarization—three years ago. Successive iterations of that feature, alongside newer capabilities like multimodal summaries (patent pending), classifier explanations, and alert confirmation, leverage a sophisticated mix of the latest models.
While LLMs excel in features like chat summarization and multimodal analysis, our rigorous testing, consistent with findings from LLM vendors like OpenAI, shows that custom-tuned smaller models often outperform LLMs for specific classification tasks. They are more cost-effective, easier to debug, and highly efficient.
As LLMs generalized into image and video modalities, we found that multimodal models, specifically Vision Language Models (VLMs) began competing with the state-of-the-art object detection models (like YOLO variants, EfficentNet and ResNet) that we had evaluated and utilized in the past. The added advantage of VLMs is their ability to use natural language labels during training, making it much easier to expand our platform into new object detection or image classification categories. Consequently, we integrated these advanced visual models directly into our image processing pipelines and ultimately used them for some of our use cases. Furthermore, because these smaller multimodal models excel at generating image captions, we integrated them to allow compliance teams to run text-based queries over visual descriptions.
We currently use LLMs to generate explanations of our classifier detections for reviewers and to create another perspective on those detections as described in the Alert Confirmation section below.
Context Expansion
Over time, we have expanded our machine learning classifier capabilities to accommodate highly flexible context lengths. Because embeddings and language models typically evaluate data at the sentence level, we engineered post-processing logic to extend classifications to paragraph and block levels. Recognizing that corporate chat logs are uniquely fragmented, we added time-scale capabilities to summarize interactions across user-defined intervals over multiple independent records. This evolved into our multimodal behavioral analysis, where compliance teams can track risks, patterns, and subtle behaviors that develop slowly over months or even years.
Needle in a Haystack: Alert Confirmation
Detecting risk is inherently difficult. To combat the 0.1% “needle in a haystack” dilemma where traditional lexicons fail, we engineered Alert Confirmation Analysis. Our machine learning classifiers, combined with Alert Confirmation Analysis, helps us minimize false positives while maintaining high recall. This framework combines our core machine learning classifiers with a global evaluation layer, expanding the context of analysis from an isolated sentence to the entire text of a record.
Beyond providing a localized decision, rationale, and confidence value for each classifier detection, Alert Confirmation Analysis evaluates the record as a whole. It delivers an overall risk score, a comprehensive rationale, and the segments of text supporting that rationale. It provides a crucial safety net in instances where a specific classifier might be turned off or a highly nuanced risk is scattered across a file. Compliance teams can use these validated scores to automatically prioritize their review queues—ensuring the highest-risk communications are audited first, while low-risk records can be safely sampled or automatically closed.
By providing reviewers with context, confidence scores, and rationales, we turn overwhelming data streams into actionable intelligence, prioritizing the most critical risks for human review.
Conclusion
From our early work in enhanced lexicons to our current modern ensembles of multilingual language models and multimodal analysis with VLMs, Theta Lake remains committed to delivering accurate, reliable, and interpretable compliance solutions. We’ve been solving the twin challenges of generalization and context awareness. We don’t just detect risks better or faster: we give compliance teams the precise context and verified rationale they need to act on them.
