Skip to navigation
Agentic Reasoning EngineHow it works

Models

View as Markdown

The assistant solves many different problems, so it runs on a collection of large and small machine learning models. Moveworks evaluates and picks each one for the job it does. The models don’t change based on your conversations. Moveworks improves them on a regular release cycle, using reviewed usage data and feedback.

The models the assistant uses

Some examples:

  1. Foundation models, such as GPT 5.4 and 5.5
    • Reason and plan actions from the user’s request
    • Call tools to fulfill the request
    • Summarize results for the user
  2. Task-specific discriminative models
    • FT-LangDetection: low-latency detection of the user’s language
    • FT-Roberta: handoff classification and entity recognition
  3. Task-specific generative models
    • FT-Flan-T5: toxicity judgment for both user input and assistant output
    • FT-Flan-T5: file search relevance judgment
    • FT-M2M100: translation of resources such as knowledge articles and forms

There are many more.

You choose the version of the reasoning model through the Frontier, Standard, or Basic track in the Reasoning Engine Upgrade Program.

Foundation models vs. task-specific models

Foundation models are the largest models available, by training data and parameter count, and they can do almost any language task, from coding and reasoning to summarization. That makes them very capable, but they have higher latency and cost, and they are not always the most reliable choice for a narrow task that needs high precision.

Task-specific models are smaller, cheaper, faster, and easier to control. Each is trained for one kind of task, such as entity recognition. They cover a subset of what foundation models can do, but at higher quality, higher speed, or lower cost. So for each task, the assistant uses the model that fits it best.

Redundancy

Moveworks routes model traffic through an LLM Gateway: a single egress point that sends each request from other Moveworks services to the best model for the task. That can be one of Moveworks’ fine-tuned models or a third-party LLM from vendors such as Azure OpenAI and OpenAI.

The gateway routes by provider. It can prioritize requests by latency needs and by regional or data-residency requirements, and it recovers automatically when a third-party endpoint is down or throttled. No single model provider is a single point of failure for the product.

How models are evaluated

Moveworks evaluates models before they ship:

  1. Evaluation datasets. Curated to cover a broad range of assistant use cases, so a model is checked across the scenarios it will see and gaps show up.
  2. Several kinds of evaluation, so overall quality keeps improving:
    • End-to-end evaluation tests the whole experience from start to finish.
    • Component evaluation tests specific parts, such as tool filtering, tool selection, and argument filling.
    • Human annotator evaluation has people review outputs and interactions for nuance and added confidence in the results.
  3. Prompt tuning. Used to fix regressions found in evaluation and to keep improving the experience. Prompt changes are tested two ways:
    • In-bot testing for immediate feedback on an adjusted prompt.
    • Large-volume evaluation on full datasets to confirm improvements across many scenarios and find what to optimize next.

How the assistant improves

Moveworks updates more than the models. Releases also bring user experience, architecture, and infrastructure improvements. Most improvements come from two places:

  1. Reviewed feedback. Moveworks annotation teams and ML engineers regularly review masked and anonymized usage data to find themes to improve. Feedback from users in Microsoft Teams or Slack, support tickets, and the Community also points to improvements for specific use cases.
  2. Targeted metric work. Moveworks monitors metrics such as latency, error rates, and response rates, and invests where it improves the platform across all use cases.

Techniques Moveworks uses for targeted improvements:

  • User simulation, which runs the system end to end as a real user would and measures overall performance
    • User behavior model
    • Goal-oriented simulation
  • Synthetic environment studies, which recreate environments to see how tools and the overall system behave under different conditions
    • Tool payload simulation
    • Stateful system simulation
  • Model-based evaluation, which speeds up and automates assessment alongside expert human annotators
    • Evaluation guidelines and prompts
    • Calibrating an LLM as a judge or juror

Does chatting with the assistant retrain it?

No. Nothing a user says or rates in chat changes the models.

There is no automated path from chat back into the models, because that can’t be done safely today (see below). Instead, Moveworks reviews randomly sampled usage data to find patterns and fix issues that come up often.

When the assistant answers, it uses the current conversation, your organization’s content and tools, and the user attributes your admin maps. See Conversation History and Personalization.

Within a conversation, the assistant follows corrections a user gives it, because earlier messages are part of what it reads before each reply. Once those messages fall outside the lookback window (24 hours by default, set by your admin), it no longer sees them. See Conversation History.

Why the assistant doesn’t learn from conversations in real time

Enterprise AI assistants don’t “learn” the way many people picture. Saying an assistant learns often suggests its models keep updating their own parameters over time. In practice, most AI assistants, including ChatGPT, Microsoft’s assistant, and the Moveworks assistant, do not train their LLMs live during conversations.

There are good reasons not to train models live on user conversations and feedback:

  1. Privacy and confidentiality. Live training risks leaking sensitive information, such as personal data, proprietary business information, or trade secrets, into the model.
    • If a model is trained on sensitive data by accident, that data could leak from one user to another, including between employees of the same Moveworks customer, or break regulations like GDPR or HIPAA.
    • Example: a user asks for a summary of a company document, finds it poor, and gives it a thumbs down. They say “try again,” get a better summary, and give it a thumbs up. They might expect the model to learn from that right away. But the summary might contain information the user didn’t realize was private to their team, client, department, or organization. If the preferred summary were trained into a model immediately, that information would be at risk. Even if the user only meant to teach it how to handle that kind of data, the model could absorb the contents too. That is too risky to leave to automation, especially automation that changes model parameters in real time.
  2. Hard to take back. Once private data is trained into a model, getting the model to “unlearn” it is very hard (see the footnote).
    • The standard way to remove learned data is to delete the model and restart from a checkpoint saved before that data was trained in. Until the model is retrained without the data, there are two bad options: roll back to the older checkpoint right away and lose quality, or keep the current model live and risk exposing the data someone asked to remove. It is far better to have people review training data so the wrong data never gets in.
  3. Data poisoning. Live training on user input widens the attack surface for data poisoning. Without people curating and checking training data, it is much easier for bad actors to manipulate a model’s behavior.
  4. Quality control. Without oversight, live training can pick up wrong, biased, or inappropriate content from user input. That degrades the assistant and produces incorrect or unreliable answers, which erodes trust.
  5. Regulatory compliance. Many customer agreements are subject to strict rules on data handling and retention. Live training can conflict with legal requirements such as GDPR’s right to be forgotten.
  6. Stability and consistency. Constant automatic updates can change behavior unexpectedly. It is better to leave a production model alone until thorough evaluation shows a newer version is ready, and any notable changes are known and communicated.
  7. Compute cost. Live training needs a lot of compute, more than any vendor can afford.

Footnote: why unlearning is hard. Machine unlearning, the work of efficiently removing specific data from a trained model, is an open research area with no definitive solution yet (see Cao & Yang, 2015; Bourtoule et al., 2019). That makes it hard to guarantee sensitive information can be reliably erased once learned, short of throwing the model out.