How does the Reasoning Engine handle large responses?

The Reasoning Engine uses a graduated approach to process plugin responses depending on their size.

Small responses (under ~7K tokens)

When a plugin response is small enough, the Reasoning Engine processes it directly in its context window. It can semantically understand, summarize, and reason over the data without any additional processing.

Large responses (~7K tokens and above, up to 100MB)

When a plugin response exceeds approximately 7K tokens, the Reasoning Engine automatically routes the data to Structured Data Analysis (SDA), which is built on the Code Interpreter. The response is stored as a variable, and the Reasoning Engine writes and runs code to analyze, filter, and extract the information the user needs.

This means plugins can return large datasets (up to 100MB) and the Reasoning Engine will still be able to work with the data.

Final response storage limits

Separately from plugin response processing, the system compresses conversation records when storing them. The final rendered bot response (text and cards) is compressed and stored in the chat history, which has an internal limit of approximately 400KB post-compression. In practice, this is rarely an issue because the Reasoning Engine’s final response to the user is typically much smaller than the raw plugin data it processed. This limit does not apply to plugin input sizes and is orthogonal to SDA’s 100MB payload support.

Recommendations

If you’re building a plugin that will return a lot of data, we recommend the following.

Plugin Selection

1. Describe the response schema: Always mention all fields from the returned data in the plugin description.

   • ❌ GetSalesforceCustomers: returns data from customers in Salesforce.
   • ✅ GetSalesforceCustomers: returns account name, owner, and renewal data for customers in Salesforce.

   This will ensure more reliable plugin selection for all the different use cases the data may support.

2. One plugin, one data source: Multiple overlapping plugins on the same dataset can create inconsistent experiences.

   • ❌ GetOpenAccountsPlugin, GetEnterpriseAccountsPlugin, GetProspectAccountsPlugin
   • ✅ GetSalesforceAccountsPlugin, GetSalesforceOpportunitiesPlugin

Response Structure

1. Use friendly names. Drop the business system default names, and rename fields to friendly, intuitive names that are easy for the model to understand. This will also make the data more friendly when it is exposed back to the user in a message, citation, or future medium.
2. Flatten your data. When possible, flatten your data into tabular fields, avoiding a nested JSON structure. This makes the data easier to work with resulting in higher model reliability.
   • Bad Example

     1
     {
     2
         "people": [
     3
             {
     4
                 "name": "John Doe",
     5
                 "details": {
     6
                     "age": 30,
     7
                     "city": "New York"
     8
                 }
     9
             },
     10
             {
     11
                 "name": "Anna Smith",
     12
                 "details": {
     13
                     "age": 25,
     14
                     "city": "London"
     15
                 }
     16
             },
     17
         ]
     18
     }
   • Good Example

     1
     {
     2
         "people": [
     3
             {
     4
                 "name": "John Doe",
     5
                 "age": 30,
     6
                 "city": "New York"
     7
             },
     8
             {
     9
                 "name": "Anna Smith",
     10
                 "age": 25,
     11
                 "city": "London"
     12
             }
     13
         ]
     14
     }

UX Recommendations

1. Disable the Activity Confirmation Policy. Given the goal is to simply fetch data, unless it is computationally expensive or you have tight rate limits, we don’t recommend getting user confirmation first. This creates needless friction for your users.
2. Add a URL field. This allows the model to link the specific business system record in the response.

   1
   LinkToAccount: "$CONCAT(['https://moveworks.lightning.force.com/lightning/r/Account/', item.Id, '/view'], '')"
3. Avoid Analysis Instructions. Unless you see plugin-specific behavior that is not intended, avoid adding instructions about how the model should analyze data (e.g. “please fuzzy match Telecam to Telecom”). This will likely be ignored and could negatively impact performance.
4. Configure Citations: if you want business records to be cited, you need to add id or friendly_id so that each record can be individually cited (instructions).

Reducing the original size

If you’re still running into issues with the size of your plugin response, consider the following

1. Schema Pruning. Eliminate fields from a data structure using the MERGE statement from our Data Mapper.
2. Attribute Compression. Reduce the length of a text field using the summarize text action
3. Record Filtering. Remove records from your list.
   1. Pre-filtering. You can collect slots or use meta info to reduce the number of records you fetch from an external system.
      1. Retrieve time series between a start_date & end_date
      2. Constrain records that are assigned_to the current user
      3. Fetch records where a numeric attribute is greater than some value (e.g. accounts above 500 users)
         You can define multiple of these inputs on the same plugin & mark the inputs as optional.
   2. Post-filtering. If the system doesn’t support API-based filtering, you can also filter the data inside your plugin’s process with DSL FILTER expressions