評估成效

Document AI 會產生精確度和召回率等評估指標，協助您判斷處理器的預測成效。

系統會透過處理器 (預測) 傳回的實體與測試文件中的註解進行比對，產生這些評估指標。如果處理器沒有測試集，請先建立資料集，然後標記測試文件。

執行評估作業

每次訓練或進階訓練處理器版本時，系統就會自動執行評估作業。

您也可以手動執行評估。如果您修改了測試集，或要評估預先訓練的處理器版本，就必須執行這項操作，才能產生更新的指標。

網路使用者介面

  1. 前往 Google Cloud 控制台的「處理器」頁面，然後選擇您的處理器。

    前往「處理器」頁面

  2. 在「Evaluate & Test」(評估與測試) 分頁中，選取要評估的處理器「版本」，然後按一下「Run new evaluation」(執行新的評估)。

完成後，頁面會顯示所有標籤和每個個別標籤的評估指標。

Python

詳情請參閱 Document AI Python API 參考文件。

如要向 Document AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。


from google.api_core.client_options import ClientOptions
from google.cloud import documentai  # type: ignore

# TODO(developer): Uncomment these variables before running the sample.
# project_id = 'YOUR_PROJECT_ID'
# location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'
# processor_id = 'YOUR_PROCESSOR_ID'
# processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'
# gcs_input_uri = # Format: gs://bucket/directory/


def evaluate_processor_version_sample(
    project_id: str,
    location: str,
    processor_id: str,
    processor_version_id: str,
    gcs_input_uri: str,
) -> None:
    # You must set the api_endpoint if you use a location other than 'us', e.g.:
    opts = ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")

    client = documentai.DocumentProcessorServiceClient(client_options=opts)

    # The full resource name of the processor version
    # e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`
    name = client.processor_version_path(
        project_id, location, processor_id, processor_version_id
    )

    evaluation_documents = documentai.BatchDocumentsInputConfig(
        gcs_prefix=documentai.GcsPrefix(gcs_uri_prefix=gcs_input_uri)
    )

    # NOTE: Alternatively, specify a list of GCS Documents
    #
    # gcs_input_uri = "gs://bucket/directory/file.pdf"
    # input_mime_type = "application/pdf"
    #
    # gcs_document = documentai.GcsDocument(
    #     gcs_uri=gcs_input_uri, mime_type=input_mime_type
    # )
    # gcs_documents = [gcs_document]
    # evaluation_documents = documentai.BatchDocumentsInputConfig(
    #     gcs_documents=documentai.GcsDocuments(documents=gcs_documents)
    # )
    #

    request = documentai.EvaluateProcessorVersionRequest(
        processor_version=name,
        evaluation_documents=evaluation_documents,
    )

    # Make EvaluateProcessorVersion request
    # Continually polls the operation until it is complete.
    # This could take some time for larger files
    operation = client.evaluate_processor_version(request=request)
    # Print operation details
    # Format: projects/PROJECT_NUMBER/locations/LOCATION/operations/OPERATION_ID
    print(f"Waiting for operation {operation.operation.name} to complete...")
    # Wait for operation to complete
    response = documentai.EvaluateProcessorVersionResponse(operation.result())

    # After the operation is complete,
    # Print evaluation ID from operation response
    print(f"Evaluation Complete: {response.evaluation}")

取得評估結果

網路使用者介面

  1. 前往 Google Cloud 控制台的「處理器」頁面，然後選擇您的處理器。

    前往「處理器」頁面

  2. 在「評估與測試」分頁中，選取要查看評估結果的處理器「版本」。

完成後，頁面會顯示所有標籤和每個個別標籤的評估指標。

Python

詳情請參閱 Document AI Python API 參考文件。

如要向 Document AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。


from google.api_core.client_options import ClientOptions
from google.cloud import documentai  # type: ignore

# TODO(developer): Uncomment these variables before running the sample.
# project_id = 'YOUR_PROJECT_ID'
# location = 'YOUR_PROCESSOR_LOCATION' # Format is 'us' or 'eu'
# processor_id = 'YOUR_PROCESSOR_ID' # Create processor before running sample
# processor_version_id = 'YOUR_PROCESSOR_VERSION_ID'
# evaluation_id = 'YOUR_EVALUATION_ID'


def get_evaluation_sample(
    project_id: str,
    location: str,
    processor_id: str,
    processor_version_id: str,
    evaluation_id: str,
) -> None:
    # You must set the api_endpoint if you use a location other than 'us', e.g.:
    opts = ClientOptions(api_endpoint=f"{location}-documentai.googleapis.com")

    client = documentai.DocumentProcessorServiceClient(client_options=opts)

    # The full resource name of the evaluation
    # e.g. `projects/{project_id}/locations/{location}/processors/{processor_id}/processorVersions/{processor_version_id}`
    evaluation_name = client.evaluation_path(
        project_id, location, processor_id, processor_version_id, evaluation_id
    )
    # Make GetEvaluation request
    evaluation = client.get_evaluation(name=evaluation_name)

    create_time = evaluation.create_time
    document_counters = evaluation.document_counters

    # Print the Evaluation Information
    # Refer to https://cloud.google.com/document-ai/docs/reference/rest/v1beta3/projects.locations.processors.processorVersions.evaluations
    # for more information on the available evaluation data
    print(f"Create Time: {create_time}")
    print(f"Input Documents: {document_counters.input_documents_count}