Yield Curve

The yield curve is only available after the documents are reviewed and released from the Systematic Random stage. This is the chart that is used to represent the ranking results. From this graph, you can determine your next steps. Do you want to cut off the review at a certain point and discard some of the collection as non-relevant?

In the following chart, responsiveness appears on the y-axis, and percentage of documents reviewed appears on the x-axis. Because the documents have been ranked and re-ranked, the curve illustrates that at the beginning of the project, the recall increases quickly and steadily rises. This is because Predict has ordered and fed the documents to the reviewers with the most likely relevant ones first. The review team is finding relevant documents. As the review gets farther along, there becomes a point at which the curve starts leveling off. After a high rate of recall is reached, more and more non-relevant documents start being reviewed.

The purpose of using Predict is to determine where you can cut off the review with the goal of reviewing fewer documents. At some point, when you start to notice the review team is reviewing fewer relevant documents, you can stop (or cut off) the review. Using the yield curve and the associated calculations, you can select a cutoff point and analyze the results.

In Insight Predict, the yield curve looks like this:

The red line represents what a yield curve would look like with a pure linear review. This allows you to compare the difference between using Predict and running a linear review project.

The blue line is the yield curve for the current project, after the Systematic Random stage is complete. The areas on the left and right that are gray, represent the portion of the collection that is already reviewed. Select a cutoff point by dragging the circle on the graph left or right. If you click the circle, the point will stick on that spot in the graph.

Notice the values to the right of the curve. These are calculated from the point you selected on the yield curve.

The numbers in these charts will help you to make important decisions and determine the appropriate cutoff point for your review. Typically, you will continue to review the documents that appear on the left side of the cutoff point, with the goal of reviewing fewer, but more relevant documents. Those documents that appear on the right side of the point are documents that you will likely not produce; you might choose to discard them completely.

Left of Cutoff

The numbers that appear in the Left of Cutoff section of the page will change, based on your selected cutoff point. As you can see, when you move the point, the numbers change.

Update Cost Variables

Select this link to change the cost variables from the defaults. A dialog box will open and you can change the information for the number of reviewers, docs per hour, hours worked per day and the cost per document. If you change any of these, it will store the information you input so you can use it again.

Right of Cutoff

The information presented in this chart allows you to see the cost and time associated with the documents you choose to be on the right side of the yield curve and are usually the documents that you choose not to review. Use the slide bar to change the cutoff and see the difference in cost.

The information tracked here is similar to the information tracked in the Left of Cutoff section of the page, but it pertains to the documents that appear on the right side of the cutoff.

These two calculations are different:

This figure is similar to the Cost Per Relevant figure from Left of Cutoff, but the number of relevant documents on the right should be sparse. Therefore, the irrelevant-to-relevant ratio should be high. For example, if the irrelevant-to-relevant ratio is 100:1, you pay for reviewing 100 documents for every one relevant document.

If there is one relevant document in every 100, and the cost for reviewing each document is $3, then the cost per relevant document is $300. So, you can see that it is an exorbitant cost to review all of the non-relevant documents just to get to the next relevant one. It is better to exclude the right side documents from review and production.

the right side documents from review and production.

Not Ranked Documents

The number of Not Ranked documents appears in the upper right corner, above the Left of Cutoff chart. It shows the number of documents (and percentage of the population) that was excluded from the ranking process. Documents might not be ranked for a variety of reasons, including file size and document content (e.g. - empty documents, documents with only numbers, documents greater than 10MB of text), are excluded from the process completely.

Click on the Not Ranked link to bring back the non-ranked documents in a table. You can, and should, review those independently.

Rejecting the Yield Curve (Continue Ranking)

After the Systematic stage is complete and all documents have been reviewed and released, you determine whether to accept or reject the yield curve.

Productivity and Coding Distribution Reports

Additional reports are available to track productivity and coding distribution.

To open the Productivity Report, click Reports in the left panel. You can customize the report by selecting all stages or by selecting just one stage, such as the QC stage or the Systematic Random stage.

Click Coding Distribution to see the decision field on which the project is based.