Audit and monitoring

​

What gets logged

​

Agent executions

• Execution metadata — Agent ID, version, executing user, workflow ID (if applicable), timestamp
• Input and output — User input to the agent, agent’s final response
• Model usage — Provider, model name, tokens consumed (prompt and completion), cost
• Duration — Total execution time, time spent in LLM calls vs. tool calls
• Outcome — Success, failure, or timeout status

​

Tool calls

• Tool metadata — Tool name, MCP server, version
• Parameters — Input parameters passed to the tool (sanitized to remove credentials)
• Outcome — Tool response or error message
• User context — Which user’s credentials were used (for OAuth-based tools)
• Duration — Time spent in the tool call

​

Data access

• Knowledge base queries — Which knowledge base, query text, results returned, user
• Database queries — Database connection, query (sanitized), rows affected, user
• File access — File path, operation (read/write), user, timestamp
• External API calls — API endpoint, HTTP method, status code (credentials removed)

​

Authentication events

• Successful logins — User, authentication method (SSO, username/password, API key), source IP, timestamp
• Failed login attempts — Username, authentication method, source IP, failure reason, timestamp
• Logouts — User, session duration, timestamp
• Token refreshes — OAuth token refreshes, user, provider (Google, Microsoft, etc.)
• API key usage — Every API request with key ID, endpoint, user, timestamp

​

Configuration changes

• Role changes — User granted or revoked from a role, by whom, timestamp
• Provider updates — Provider added, modified, or deleted, which fields changed, by whom
• Project modifications — Project created, settings changed, deleted, by whom
• Agent and workflow changes — Version control for agent and workflow definitions, who made each change
• Policy changes — Organization policies, quotas, or settings modified, by whom

​

Workflow executions

• Workflow metadata — Workflow ID, definition version, triggering user or API key, timestamp
• Activity executions — Each activity (agent, tool, human task) with input, output, duration, status
• Retry history — Failed activities with retry attempts, backoff intervals, eventual outcome
• Human task events — Task assigned, completed, or escalated, assignee, completion time
• Workflow outcome — Success, failure, timeout, or cancellation

​

Viewing audit trails

​

Admin Portal — Platform-wide logs

• All organizations, users, and authentication events
• Platform-level provider configurations and changes
• Cross-organization metrics and usage trends
• Security events like failed logins and unauthorized access attempts

1. Log into the Admin Portal
2. Navigate to “Audit Logs” in the sidebar
3. Filter by event type, date range, user, or organization

​

Studio — Project-level logs

• Agent and workflow executions within the project
• Tool calls made by agents in the project
• Knowledge base queries and data access
• Project configuration changes and role assignments

1. Log into Studio
2. Select the project from the sidebar
3. Navigate to “Execution History” or “Audit Logs” in the project menu
4. Filter by agent, workflow, date range, or user

​

Filtering and search

• By user — See all actions by a specific user (useful for offboarding reviews)
• By event type — Filter to only agent executions, or only configuration changes
• By date range — Narrow down to a specific time window (e.g., “last 7 days”)
• By resource — See all events related to a specific agent, workflow, or tool
• By outcome — Filter to only failures or errors

​

Exporting logs

​

Execution monitoring

​

Real-time dashboard

• Active workflows — Currently running workflows with status (running, waiting, failing)
• Activity-level status — Which activity is currently executing, how long it’s been running
• Resource consumption — Tokens consumed, cost incurred, API calls made (updated in real-time)
• Queued executions — Workflows waiting to start (if rate limiting or concurrency limits apply)

​

Activity-level details

• Execution timeline — Visual representation of which activities have completed and which are running
• Activity inputs and outputs — For completed activities, see the input data and output results
• Retry attempts — If an activity failed and is retrying, see the retry count and backoff interval
• Estimated completion — Based on historical data, an estimate of when the workflow will complete

​

Error and failure tracking

• Failed workflows — Workflows that terminated due to errors, with the failing activity and error message
• Retrying activities — Activities that failed but are being retried, with current attempt count
• Timeout warnings — Activities approaching their timeout threshold
• Rate limit warnings — Activities waiting due to rate limiting on external APIs

​

Integration with monitoring systems

​

Structured log output

{
  "timestamp": "2025-12-15T14:32:18Z",
  "event_type": "agent_execution",
  "user_id": "usr_abc123",
  "user_email": "[email protected]",
  "organization_id": "org_xyz789",
  "project_id": "proj_def456",
  "agent_id": "agent_ghi789",
  "agent_name": "Customer Support Agent",
  "model": "gpt-4",
  "tokens_prompt": 1250,
  "tokens_completion": 380,
  "cost_usd": 0.0234,
  "duration_ms": 3400,
  "outcome": "success"
}

​

Log forwarding

• Syslog — Forward logs via syslog protocol to a central log server
• HTTP endpoint — POST logs to a webhook or log ingestion API
• File output — Write logs to a file for collection by a log shipper (Filebeat, Fluentd, etc.)

​

SIEM integration

1. Forward logs — Use syslog or HTTP forwarding to send audit logs to your SIEM (Splunk, Sentinel, etc.)
2. Create detection rules — Set up rules to detect anomalous patterns (e.g., “User with 10+ failed logins in 5 minutes”)
3. Correlate with other systems — Join MagOneAI logs with logs from your identity provider, network, and other systems
4. Alert on security events — Trigger alerts for suspicious activity (unusual token usage, unauthorized access attempts)

• Compromised credentials — Detect API keys or OAuth tokens used from unexpected IP addresses
• Insider threats — Detect users accessing data outside their normal patterns
• Compliance violations — Detect access to sensitive data without proper authorization
• Brute force attacks — Detect repeated failed login attempts

​

Custom monitoring dashboards

1. Export historical data — Use the audit log export feature to download data as CSV or JSON
2. Import to your BI tool — Load the data into Tableau, Power BI, Looker, or Grafana
3. Build dashboards — Create visualizations for metrics like executions per day, cost per project, error rates, etc.
4. Schedule refreshes — Set up automated exports and dashboard refreshes for up-to-date metrics

​

Cost and token usage tracking

​

Token consumption tracking

• Prompt tokens — Tokens in the input to the model (system prompt, user input, context)
• Completion tokens — Tokens in the model’s output
• Total tokens — Sum of prompt and completion tokens
• Model — Which model was used (gpt-4, claude-3-opus, etc.)

​

Cost calculation

• Per-token pricing — For models with separate prompt and completion pricing (e.g., GPT-4)
• Per-request pricing — For models with fixed request costs (e.g., some image generation models)
• Currency conversion — Costs normalized to USD for consistent reporting

​

Cost breakdowns

• Organization — Total spend across all projects in the org
• Project — Spend within a specific project
• Agent — Which agents are most expensive to run
• Workflow — Cost of individual workflow executions
• User — Which users are triggering the most expensive executions
• Provider and model — Spend per LLM provider (OpenAI vs. Anthropic) and model (GPT-4 vs. GPT-3.5)

​

Usage trends and forecasting

• Trend charts — Visualize token usage and costs over time (daily, weekly, monthly)
• Forecasting — Project future costs based on current usage trends
• Anomaly detection — Alert when usage spikes unexpectedly (e.g., 10x increase in a single day)

​

Budget alerts and quota enforcement

​

Compliance and retention

​

Log retention policies

• Default retention — 90 days for execution logs, 1 year for authentication and configuration change logs
• Custom retention — Extend retention to meet your compliance needs (e.g., 7 years for financial services)
• Automatic archival — Old logs are automatically archived to cold storage (S3, Azure Blob, etc.)
• Deletion policies — Logs older than the retention period are automatically deleted

​

Immutability and tamper-evidence

• Append-only storage — Logs can be written but never modified or deleted (except by automated retention policies)
• Cryptographic hashing — Each log entry is hashed and chained to previous entries, making tampering detectable
• Access logging — Access to audit logs is itself logged, creating a complete chain of custody

​

Audit log exports for compliance

1. Define the audit period — E.g., “All activity from January 1 - December 31, 2025”
2. Filter to relevant events — E.g., “All access to customer data” or “All role changes”
3. Export to CSV or PDF — Generate a human-readable report with timestamps, users, and actions
4. Provide to auditors — Share the export with your compliance team or external auditors

​

Advanced monitoring

​

Distributed tracing

• Trace propagation — Every workflow execution generates a trace ID that propagates through all activities
• Span details — Each agent execution, tool call, and LLM request is a span with timing and metadata
• Trace export — Traces can be exported to Jaeger, Zipkin, or other tracing backends

​

Metrics and observability

• Execution counts — Total workflows executed, success rate, failure rate
• Latency histograms — P50, P95, P99 latency for agent executions
• Token usage — Total tokens consumed per minute, per project, per model
• Cost metrics — Total spend per minute, per project, per organization

​

Custom alerts

• Failure rate alert — Trigger when workflow failure rate exceeds 10% over 5 minutes
• Cost spike alert — Trigger when hourly spend is 3x the daily average
• Security alert — Trigger on any failed authentication attempt from an unusual IP
• SLA alert — Trigger when workflow execution time exceeds an SLA threshold

​

Troubleshooting with audit logs

​

Example: Debugging a failed workflow

​

Example: Investigating unusual costs

​

Example: Security investigation

​

Best practices

​

Next steps