Policy Analyzer

Background

Legal frameworks have different requirements for an organization’s policies. For example, NIST RMF requires you to address how AI system incidents are documented and addressed; while the EU AI Act has provisions concerning how interactions with AI systems are disclosed. It is important to review your organization’s existing policies and identify gaps relative to frameworks that you need to comply with. Trustible’s Policy Analyzer is an automated, AI-powered tool for conducting this review.

How it works

The Policy Analyzer takes as input a set of Policy Documents and a legal framework of choice. For each legal framework, we have curated a set of questions that must be answered for each section/article of that framework (similarly to the Guiding Questions); the Analyzer reviews your documents for answers to these questions and creates a report that highlights which answers are missing in your documents. After reviewing the report, you can iterate on your policies and rerun the the Analyzer.

Getting Started

1. Head over to the Policies tab and click on Upload to AI Analyzer (or click here:(https://app.trustible.ai/ml/policy/upload_policy/)

1. Select the Policy that you would like to Analyze

   1. For ‘Select Existing Policy’, only Policies with content (text or files will be shown).
   2. For Upload Files, the documents will be used for analysis, but will not be saved elsewhere in the platform.
   3. COMING SOON: Ability to analyze multiple policies and frameworks at the same time.

   

2. When you click ‘Run Report’, you will be taken to the Report list view. The Analyzer will start running in the background, feel free to close this tab. The analysis will be ready in several minutes. You can find this tab at: https://app.trustible.ai/policy/reports/

   1. NOTE: A “Notification” system is not currently implemented, so you will have to come back to this page to check on the progress.

      

3. When the report is ready, you will be able to click on the name of the report (e.g Policy Analysis 27) to review your framework readiness.

   1. The Report will show you how many articles are complete and how many need review. An article is considered complete if it answers all of our questions associated with it. “Answered” is calculated using an AI-powered method that analyzes your text.

      

   2. You can open the tab associated with each Article to review the questions. We recommend starting your review with the questions that have not been answered according to the analyzer, but reading through all the questions and answers is useful for a final review.

4. We collect feedback both on the questions and on the AI Analyzer’s answers. In the detailed view for every article, you have the option to:

   1. Ignore specific questions - by clicking the “crossed out eye”. This will remove the question from the completeness calculation. Note: the ignore will only be applied in the context of this specific Report, it will not be copied to other reports.

   

   b. Upvote or Downvote specific answers - this feedback will be used for analytics and to help us improve models over time. These votes are not used for the completeness calculation - if you disagree with the AI’s assessment, you can Ignore the question.

   

   1. [Custom Policies Only] In addition to doing the review, you can use the report to create a permanent link between the Policy and the Framework Article. Simply click the “plus” icon in the row associated with the article.

      

      By creating this link, you will create an updated completeness calculation in the Frameworks portion of the platform.