Monitor with Traces

Learn how to configure and use the Traces feature to gain visibility into how your AI agents process messages, invoke tools, and verify response quality.

What You'll Accomplish

By the end of this guide, you will:

• Enable traces at the site level
• Configure detailed traces for tool invocations
• Set up access control for trace visibility
• Override trace settings per chat app
• Understand different trace types
• Use traces for debugging and monitoring

Prerequisites

• A running Pika installation
• Access to pika-config.ts for site-level configuration
• Understanding of your user types and roles
• Admin or developer access for debugging

Understanding Traces

Traces provide visibility into the LLM's reasoning process, showing step-by-step how it processes user messages and generates responses.

Trace Types

Orchestration Traces:

• Step-by-step reasoning process
• Decision-making rationale
• Planning and strategy selection
• High-level reasoning steps

Failure Traces:

• Specific error messages
• Root cause analysis
• Failed execution paths
• Debugging information

Tool Invocation Traces (Detailed Traces):

• Tool names and versions
• Input parameters (when detailed traces enabled)
• Output results
• Execution timing

Response Verification Traces:

• Verification grades (A, B, C, F)
• Accuracy assessments
• Auto-reprompt decisions
• Quality reasoning

Step 1: Enable Basic Traces

Configure traces in your pika-config.ts to enable them site-wide.

Location: apps/pika-chat/pika-config.ts

export const pikaConfig: PikaConfig = {
    siteFeatures: {
        traces: {
            enabled: true,
            userTypes: ['internal-user'],
            userRoles: ['developer', 'support-agent'],
            applyRulesAs: 'or' // User needs userType OR userRole
        }
    }
};

Set User Types/Roles

Setting enabled: true alone is not sufficient. You must specify userTypes or userRoles for users to see traces. Without these, traces are disabled for everyone.

Configuration Options

Property	Type	Description
enabled	boolean	Enable the traces feature
userTypes	string[]	User types that can see traces
userRoles	string[]	User roles that can see traces
applyRulesAs	'and' | 'or'	How to combine userTypes and userRoles
detailedTraces	AccessRules	Additional access rules for detailed traces

Step 2: Enable Detailed Traces (Optional)

Detailed traces show tool invocation parameters and are typically restricted to admins and developers.

export const pikaConfig: PikaConfig = {
    siteFeatures: {
        traces: {
            enabled: true,
            userTypes: ['internal-user'],
            detailedTraces: {
                enabled: true,
                userTypes: ['internal-user'],
                userRoles: ['pika:content-admin', 'developer'],
                applyRulesAs: 'or' // Internal users OR specific roles
            }
        }
    }
};

Why Separate Detailed Traces?

Detailed traces may contain:

• Sensitive parameter data
• Internal implementation details
• System architecture information
• Performance metrics

Restricting access ensures only authorized users see this sensitive information.

Step 3: Configure Access Control

Set up fine-grained access based on your security requirements.

Internal Users Only

traces: {
    enabled: true,
    userTypes: ['internal-user']
}

Specific Roles

traces: {
    enabled: true,
    userRoles: ['developer', 'support-lead', 'qa-engineer']
}

Combined Rules (AND Logic)

traces: {
    enabled: true,
    userTypes: ['internal-user'],
    userRoles: ['developer'],
    applyRulesAs: 'and' // Must be internal AND developer
}

Tiered Access

traces: {
    enabled: true,
    userTypes: ['internal-user'], // All internal users see basic traces
    detailedTraces: {
        enabled: true,
        userTypes: ['internal-user'],
        userRoles: ['pika:content-admin'], // Only admins see detailed traces
        applyRulesAs: 'and'
    }
}

Step 4: Override Per Chat App (Optional)

Individual chat apps can customize trace settings.

Disable for Specific Chat App

const simpleChatApp: ChatApp = {
    chatAppId: 'customer-faq',
    title: 'Customer FAQ',
    // ... other properties
    features: {
        traces: {
            featureId: 'traces',
            enabled: false // Disable traces for this app
        }
    }
};

More Restrictive Access

const sensitiveChatApp: ChatApp = {
    chatAppId: 'financial-advisor',
    title: 'Financial Advisor',
    // ... other properties
    features: {
        traces: {
            featureId: 'traces',
            enabled: true,
            userTypes: ['internal-user'],
            userRoles: ['finance-admin'], // More restrictive
            applyRulesAs: 'and',
            detailedTraces: {
                enabled: true,
                userTypes: ['internal-user'],
                userRoles: ['pika:content-admin'],
                applyRulesAs: 'and'
            }
        }
    }
};

Complete Override Required

When overriding traces, you must provide ALL settings including both basic and detailed trace configuration. Overrides completely replace site-level settings.

Step 5: Deploy and Test

Deploy Your Configuration

# If using local development
cd apps/pika-chat
pnpm run dev

# If deploying to AWS
cd services/pika
pnpm run deploy

Test Trace Display

1. Log in as a user with trace access
2. Start a chat session
3. Ask a question that triggers tool use
4. Observe traces appear at top of response:
   • Expandable trace sections
   • Grouped by type
   • Syntax highlighted JSON

Verify Access Control

Test with different user types:

Trace Storage and Lazy Loading

With the Strands converse implementation (siteFeatures.strandsConverse.enabled), assistant-message traces are stored in S3 instead of in the DynamoDB message item.

• Trace object: written gzipped to the Pika S3 bucket at traces/{userId}/{sessionId}/{messageId}.json.gz.
• System prompt: stored once per session as system-prompt-{hash}.txt.gz and referenced from each trace instead of being repeated.
• Message item: carries a tracesS3 pointer to the object.
• Fallbacks: if the S3 write fails, the traces are stored inline, capped to fit the 400 KB DynamoDB item limit. If that write also fails, the message is stored without traces so the reply is not lost.
• Legacy messages: messages with inline traces continue to read as before.

The messages API returns hasOffloadedTraces: true in place of the pointer. The chat UI loads the traces when the user expands Answer Reasoning, using GET /api/trace/[chatAppId]/[sessionId]/[messageId] on pika-chat, which calls the chatbot API at GET /api/chat/{sessionId}/messages/{messageId}/traces (a share/entity variant exists for shared sessions). Each message shows one of these states:

State	Display
Loading	"Loading reasoning..."
Error	"Couldn't load the reasoning for this answer." with a Retry button
Empty	"No reasoning was recorded for this answer."

Admin reads (getChatMessagesAsAdmin with includeTraces=true) hydrate offloaded traces in parallel, up to about 5 MB per response. Messages past that limit stay lazily loadable.

Bucket Lifecycle and Permissions

• A traces/ lifecycle rule expires objects after 400 days. Keep this longer than your message retention. Adding the rule replaces any lifecycle rules that were set on the bucket outside CDK.
• The converse-strands Lambda has s3:PutObject on traces/*. The chatbot API and the insights runner have s3:GetObject on traces/*. The chatbot API Lambda also receives PIKA_S3_BUCKET.
• The pika-chat role is explicitly denied traces/*. The download, s3-file-content and file-upload routes reject traces/ keys with a 403.
• When a message is deleted by TTL or manually, its S3 trace object is deleted. The message-changed stream handler rehydrates offloaded traces before indexing into OpenSearch (INSERT events only), and the session-insights analyzer hydrates them before analysis.

Restricting Trace Content with a Hook

siteFeatures.traces decides who sees the traces feature. A second, code-level check decides whether a user may receive trace content (system prompt, tool inputs and outputs) at all. It lives in the sync-protected file apps/pika-chat/src/lib/custom/trace-content-access.ts:

export function mayReceiveTraceContent(user: ChatUser<RecordOrUndef> | undefined, tracesFeatureEnabled: boolean): boolean

The default returns true. When it returns false for a user:

• The trace route returns 403.
• The messages route strips traces, and so does the site-admin view.
• The live stream is redacted to label-only step frames, for both converse implementations.
• converse-strands is told through features.traces.enabled.

This hook does not cover detailedTraces, which is a UI-only setting.

apps/pika-chat/src/lib/custom/trace-content-access.ts

export function mayReceiveTraceContent(user, tracesFeatureEnabled) {
    return user?.userType === 'internal-user';
}

Label-Only Step Frames

For a turn whose trace content is withheld, the stream carries { step: { event: 'start' | 'done', id, label?, labelDone? } } frames and an activity frame instead of trace payloads. The chat UI renders them as a step list. Tool function schemas set the text with two optional fields:

functionSchema: [
    {
        name: 'lookup_item',
        description: 'Look up an item',
        userLabel: 'Looking up your item',
        userLabelDone: 'Looked up your item',
        parameters: { /* ... */ }
    }
]

userLabel is shown while the tool runs and userLabelDone after it completes. Without them, the step has no label text.

Understanding Trace Display

Trace Categories

Traces are automatically categorized and displayed:

Text Traces:

• General reasoning and rationale
• Planning and strategy
• Decision explanations

Tool Invocation Traces:

• Grouped by tool name
• Shows tool calls and results
• Parameters visible with detailed traces

Verification Traces:

• Response quality grades
• Accuracy assessments
• Auto-reprompt decisions

Failure Traces:

• Error messages
• Diagnostics
• Troubleshooting information

Trace UI Features

• Expandable Sections: Click to expand/collapse trace groups
• Syntax Highlighting: JSON and code automatically highlighted
• Grouped Display: Similar traces grouped together
• Timestamp Information: Shows when traces occurred
• Verification Badges: Quality grades displayed prominently

Integration with Other Features

With Verify Response Feature

When both traces and verify response are enabled:

siteFeatures: {
    traces: {
        enabled: true,
        userTypes: ['internal-user']
    },
    verifyResponse: {
        enabled: true,
        autoRepromptThreshold: 'C',
        userTypes: ['internal-user']
    }
}

Result: Traces show verification grades and reasoning

With Content Admin Feature

Content admins can view traces for other users' sessions:

// User with pika:content-admin role can:
// - View any user's chat sessions
// - See all traces for those sessions
// - Debug issues across all users

Use Cases

Development and Debugging

traces: {
    enabled: true,
    userTypes: ['internal-user'],
    userRoles: ['developer', 'qa-engineer'],
    detailedTraces: {
        enabled: true,
        userTypes: ['internal-user'],
        userRoles: ['developer'],
        applyRulesAs: 'and'
    }
}

Benefits:

• Debug agent behavior
• Understand tool invocations
• Identify failure patterns
• Optimize prompts

Customer Support

traces: {
    enabled: true,
    userTypes: ['internal-user'],
    userRoles: ['support-agent', 'support-lead'],
    detailedTraces: {
        enabled: false // Support doesn't need detailed traces
    }
}

Benefits:

• Understand customer issues
• Verify agent responses
• Identify problematic interactions
• Improve support quality

Quality Assurance

traces: {
    enabled: true,
    userTypes: ['internal-user'],
    userRoles: ['qa-engineer', 'quality-analyst'],
    detailedTraces: {
        enabled: true,
        userRoles: ['qa-engineer'],
        applyRulesAs: 'or'
    }
}

Benefits:

• Validate agent behavior
• Ensure quality standards
• Document test cases
• Track improvements

Testing Checklist

Verify traces work correctly:

Troubleshooting

Traces Not Appearing

• Verify enabled: true in site configuration
• Check user has required userTypes or userRoles
• Review chat app overrides (may have disabled traces)
• Ensure agent is generating trace data
• Check browser console for errors

Detailed Traces Missing

• Verify detailedTraces.enabled: true
• Check user meets detailed traces access requirements
• Confirm user has necessary roles
• Review access rule logic (applyRulesAs)

Performance Issues

• Traces can increase response display time
• Consider restricting to internal users only
• Disable detailed traces if not needed
• Use chat app overrides to disable for high-traffic apps

Debug Access Issues

// Log trace configuration
console.log('Traces Config:', pikaConfig.siteFeatures.traces);
console.log('User Type:', user.userType);
console.log('User Roles:', user.roles);

// Check if user should see traces
const hasAccess = checkTraceAccess(user, tracesConfig);
console.log('User has trace access:', hasAccess);

Security Considerations

Trace Content

Traces may contain:

• User input data
• Tool parameters and results
• Internal system information
• Performance metrics

Recommendation: Only enable for trusted internal users

Detailed Traces

Detailed traces are even more sensitive:

• API keys and credentials (if in parameters)
• Database queries
• Internal architecture details
• System vulnerabilities

Recommendation: Restrict to admins and developers only

Compliance

For regulated industries:

• Review trace content for compliance
• Consider data retention policies
• Implement audit logging for trace access
• Document who can access traces

Next Steps

• Enable Self-Correcting Responses - Add verification traces
• Set Up the Admin Site - View traces across all users
• Override Default Features - Customize per chat app

Related Documentation

• Traces Capability - Learn more about traces
• Verify Response - Response verification
• Feature Configuration Reference - Complete options

---

Powerful Debugging Tool

Traces are invaluable for understanding and debugging agent behavior. Enable them during development and for support staff to improve your AI applications.