Administration

Getting there

1. Sign in to Sortd-AI.
2. Open the navigation menu.
3. Under Manage, select Access Requests when you need to review requests to join.
4. Under Observability, select the area you need:
   • Agent Insights for an overview of AI activity.
   • Conversations for individual AI interactions and review ratings.
   • Queue health for current processing activity.
   • LLM cost for cost totals and breakdowns.
   • Repo Services for repository service activity.
   • Analyzer runs for the history of analysis runs.
5. Select Audit log to review user-initiated activity across Sortd-AI.

Some Administration areas are available only to users with the required role. If an area is not shown, ask your Sortd-AI administrator to check your access.

Key concepts

• Access Requests are requests submitted through the Request Access form. They are grouped as pending or reviewed. Reviewed requests can be approved or rejected.
• An approved request can show Invite sent, Accepted, or Invite expired. An invitation that has not been accepted can be sent again with Resend invite.
• Audit log contains user-initiated activity across Sortd-AI. You can narrow the list by date, action, user email, organization, or app, and review up to 50 entries at a time.
• Agent Insights provides an overview for a selected period: the last 24 hours, last 7 days, or last 30 days. It includes query volume, average and p95 time, LLM cost, escalations, failures, outcomes, and areas that need attention.
• Conversations lists AI interactions. It includes the query, outcome, intent, duration, cost, rating, and time. The list opens with Bad ratings selected so that items needing review are easy to find.
• Conversation turn shows the details of one interaction, including its outcome, duration, tokens, cost, revisions, response, and reviewer verdict. You can rate it Good or Bad and add an optional note.
• Queue health shows waiting, active, delayed, and failed work, plus the oldest active age. It marks a queue as backed up when it reaches the displayed depth threshold or age threshold.
• LLM cost shows the current period's totals by model and organization. It separates Platform cost from BYOP spend.
• Repo Services provides repository service health and activity. Analyzer runs shows the status, coverage, timing, warnings, and details for each run.

How to manage Administration

Use Administration as a review point rather than changing settings without checking the effect first.

1. Start with Fleet overview to see organizations needing attention. It highlights organizations near quota, past-due or unpaid billing, test access expiring within 7 days, missing billing status, and failed generation runs in the last 24 hours.
2. Use Organizations when you need to open an organization's details, members, apps, billing, or usage.
3. Use Audit log after an account-level change when you need to confirm who initiated it and when.
4. Use Agent Insights and its related areas when you need to understand AI performance, cost, failures, or processing activity.
5. Refresh data before acting on an item that may have changed since you opened it.

Only make changes that match your organization or support responsibilities. Billing, organization, platform, and partner controls may require a higher level of access than review areas.

How to review account-level work

Review activity

Open Audit log to see user-initiated activity across Sortd-AI. Use the available filters to focus on a date range, action, user email, organization, or app. Move through the results with ← Prev and Next →.

Review AI activity

Open Agent Insights and choose a time period. Review:

• Query volume and response timing.
• LLM cost and token totals.
• Escalation and failure percentages.
• How queries resolved, such as Answered, Escalated, Clarified, Action, or Timed out.
• Recent failures and items marked as an unfiled defect.

Open Conversations for a more focused review. Filter by outcome, intent, rating, or app, or search for a query. Select a conversation to open its Conversation turn details. You can export the filtered list with Export CSV.

When reviewing a Conversation turn, compare the user query, final response, timing, revisions, and response cost. Select Good or Bad, then add an optional note explaining the verdict.

Review processing and repository activity

Open Queue health to check waiting, active, delayed, and failed counts. Select Refresh for current values. A queue can be marked as backed up when waiting work or the oldest active item reaches the displayed alert threshold.

Open Repo Services to review repository service activity. Select View analyzer run history for an app to open Analyzer runs. Expand a run when it contains warnings, a failure reason, or per-service detail. Run status can be complete, running, pending, or failed.

How to use Access Requests

1. Open Access Requests.
2. Review the pending count and the requester’s name, email address, company name, and submission time.
3. For a request that should join, select Approve.
4. For a request that should not join, select Reject.
5. Use Refresh to reload the list.
6. Review the item in the reviewed list. An approved request can show Invite sent, Accepted, or Invite expired.
7. If an approved invitation has not been accepted, select Resend invite.

Approved invitations are separate from accepted membership. Ask the recipient to complete the invitation if the item remains Invite sent. If it shows Invite expired, use Resend invite when that action is available.

Tips & things to know

• Check Fleet overview first when you need a quick list of account-level items needing attention.
• Use the last 7 days in Agent Insights for a balanced operational view; switch to the last 24 hours for recent issues or the last 30 days for broader trends.
• Conversations opens with Bad ratings selected. Change the rating filter to review all ratings or another category.
• In LLM cost, Platform cost and BYOP spend are shown separately. BYOP spend is tenant-key spend and is informational in this view.
• An empty list is not necessarily an error. Conversations, Analyzer runs, and other areas can have no data for the selected period or app.
• Refresh after another administrator has approved, rejected, or resent an invitation.
• Treat AI responses and review notes as operational information. Check the original context before making a customer-facing or account-level decision.

Troubleshooting

FAQ