Extracting and Verifying Insurance Policy Details with React, Document Scanner, and Gemini AI on a Self Service Kiosk

Published: 10/01/2025

Building upon our journey of integrating hardware into web applications, we'll now enhance our document scanner example to perform a more targeted task: extracting specific information from an insurance policy and verifying it against a hypothetical backend system. This showcases a practical application of combining document scanning with the intelligent data extraction capabilities of Google Gemini.

Instead of simply summarising the entire document, we'll instruct Gemini to extract the policyholder's name and the policy ID. This extracted information will then be sent to a sample backend API for verification. This pattern is common in real-world scenarios where you need to validate information against a system of record.

Our setup remains similar: you'll need Node.js, npm (or yarn), a basic understanding of React and JavaScript, our SDK's API running at http://localhost:9094/, a connected document scanner, and access to the Google Gemini API. We'll also assume you have a sample backend API running and accessible (we'll use a placeholder URL for this).

Let's start with our src/App.js file. We'll modify the handleScanDocument function to extract the specific data using Gemini and then send it to the backend. Replace the contents of your src/App.js with the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
import React, { useState, useEffect } from 'react';

function App() {
  const [status, setStatus] = useState('Loading...');
  const [scanResult, setScanResult] = useState(null);
  const [policyHolderName, setPolicyHolderName] = useState('');
  const [policyId, setPolicyId] = useState('');
  const [verificationStatus, setVerificationStatus] = useState('');

  useEffect(() => {
    const checkStatus = async () => {
      try {
        const response = await fetch('http://localhost:9094/api/hardware/documentreader/get-status');
        const data = await response.json();
        if (data.code === '0000' && data.message === 'WORKING') {
          setStatus('Document scanner is: Ready');
        } else if (data.code === '0000' && data.message === 'NOT_FOUND') {
          setStatus('Document scanner: Not found');
        } else {
          setStatus('Error: ' + data.message);
        }
      } catch (error) {
        setStatus('Error connecting to API');
      }
    };

    checkStatus();
  }, []);

  const handleScanDocument = async () => {
    setScanResult('Scanning...');
    setPolicyHolderName('');
    setPolicyId('');
    setVerificationStatus('Not Verified');
    try {
      const response = await fetch('http://localhost:9094/api/hardware/documentreader/read', {
        method: 'POST',
      });
      const data = await response.json();
      if (data.code === '0000') {
        setScanResult(data.data);
        console.log('Document Data:', data.data);

        // Extract data using Gemini
        const extractedText = `Document Text: ${JSON.stringify(data.data.documentOcr)}`; // Send the OCR data
        try {
          const geminiResponse = await fetch('YOUR_GEMINI_API_ENDPOINT', { // Replace with your Gemini API endpoint
            method: 'POST',
            headers: {
              'Content-Type': 'application/json',
              // 'Authorization': `Bearer YOUR_GEMINI_API_KEY`, // Secure your API key! Use a backend for production.
            },
            body: JSON.stringify({
              prompt: `From the following text, extract the full name of the policy holder and the policy ID. Return the name and ID as a JSON object with keys "policyHolderName" and "policyId". Text: ${extractedText}`,
            }),
          });

          const geminiData = await geminiResponse.json();
          if (geminiData && geminiData.candidates && geminiData.candidates.length > 0) {
            try {
              const geminiResponseData = JSON.parse(geminiData.candidates[0].content.parts[0].text);
              setPolicyHolderName(geminiResponseData.policyHolderName || 'Extraction Failed');
              setPolicyId(geminiResponseData.policyId || 'Extraction Failed');

              // Send to backend for verification
              setVerificationStatus('Verifying...');
              try {
                const backendResponse = await fetch('YOUR_BACKEND_VERIFICATION_ENDPOINT', { // Replace with your backend endpoint
                  method: 'POST',
                  headers: {
                    'Content-Type': 'application/json',
                  },
                  body: JSON.stringify({
                    policyHolderName: geminiResponseData.policyHolderName,
                    policyId: geminiResponseData.policyId,
                  }),
                });

                const backendData = await backendResponse.json();
                if (backendData.isValid) {
                  setVerificationStatus('Verified');
                } else {
                  setVerificationStatus('Verification Failed');
                }
              } catch (backendError) {
                setVerificationStatus('Verification Error');
                console.error('Backend Verification Error:', backendError);
              }

            } catch (parseError) {
              console.error('Error parsing Gemini JSON:', parseError);
              setPolicyHolderName('Extraction Failed');
              setPolicyId('Extraction Failed');
            }
          } else {
            setPolicyHolderName('Extraction Failed');
            setPolicyId('Extraction Failed');
          }
        } catch (geminiError) {
          console.error("Gemini API Error:", geminiError);
          setPolicyHolderName('Extraction Error');
          setPolicyId('Extraction Error');
        }

      } else {
        setScanResult('Scan Failed: ' + data.message);
      }
    } catch (error) {
      setScanResult('Error during scan.');
    }
  };

  return (
    <div>
      <h1>Insurance Policy Verification</h1>
      <p>Status: {status}</p>
      <button onClick={handleScanDocument} disabled={status !== 'Document scanner is: Ready'}>
        Scan Insurance Policy
      </button>
      {scanResult === 'Scanning...' && <p>Scanning document...</p>}
      {policyHolderName && policyId && (
        <div>
          <h3>Extracted Information:</h3>
          <p>Policy Holder Name: {policyHolderName}</p>
          <p>Policy ID: {policyId}</p>
          <p>Verification Status: {verificationStatus}</p>
        </div>
      )}
      {scanResult && typeof scanResult === 'string' && scanResult !== 'Scanning...' && !policyHolderName && <p>{scanResult}</p>}
    </div>
  );
}

export default App;

Here’s a breakdown of the key changes:

New State Variables: We've added policyHolderName, policyId, and verificationStatus state variables to store the extracted information and the result of the backend verification.
Updated Gemini Prompt: The prompt sent to Gemini is now more specific:

"From the following text, extract the full name of the policy holder and the policy ID. Return the name and ID as a JSON object with keys policyHolderName and policyId. Text: ${extractedText}"

We are instructing Gemini to return a JSON object containing the extracted name and ID.

JSON Parsing of Gemini Response: We now parse the text content of Gemini's response as JSON:

1
2
3
const geminiResponseData = JSON.parse(geminiData.candidates[0].content.parts[0].text);
setPolicyHolderName(geminiResponseData.policyHolderName || 'Extraction Failed');
setPolicyId(geminiResponseData.policyId || 'Extraction Failed');

We use a try...catch block to handle potential errors during JSON parsing.

Backend Verification: After successfully extracting the name and ID from Gemini, we send this data to a sample backend endpoint for verification:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
try {
    const backendResponse = await fetch('YOUR_BACKEND_VERIFICATION_ENDPOINT', { // Replace with your backend endpoint
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            policyHolderName: geminiResponseData.policyHolderName,
            policyId: geminiResponseData.policyId,
        }),
    });

    const backendData = await backendResponse.json();
    if (backendData.isValid) {
        setVerificationStatus('Verified');
    } else {
        setVerificationStatus('Verification Failed');
    }
} catch (backendError) {
    setVerificationStatus('Verification Error');
    console.error('Backend Verification Error:', backendError);
}

Remember to replace YOUR_BACKEND_VERIFICATION_ENDPOINT with the actual URL of your backend API. We send the extracted policyHolderName and policyId in the request body and update the verificationStatus based on the backend's response.

UI Updates: The UI now displays the extracted Policy Holder Name, Policy ID, and the Verification Status. The raw scanResult is no longer displayed by default to focus on the extraction and verification process.

Important Considerations

Replace Placeholders: Make sure to replace YOUR_GEMINI_API_ENDPOINT and YOUR_BACKEND_VERIFICATION_ENDPOINT with your actual API endpoints.
Secure API Keys: As emphasised before, never expose your Gemini API key directly in the frontend code. Implement a secure backend to handle communication with the Gemini API.
Backend Implementation: This example assumes your backend API has an endpoint that accepts the policy holder name and policy ID and returns a JSON response with a boolean isValid property. You'll need to implement this backend logic.
Error Handling: We've added basic error handling for the Gemini API call, JSON parsing, and the backend API call. You should expand this error handling to provide more informative messages to the user.
Gemini Prompt Engineering: The effectiveness of the information extraction depends on the quality of your Gemini prompt. Experiment with different prompts to achieve the desired accuracy.
Loading States: Consider adding more granular loading states to provide better feedback to the user during the Gemini API call and backend verification process.

This enhanced example demonstrates a more complete workflow for extracting specific data from scanned documents and integrating with a backend system for verification. This pattern is highly applicable in various real-world scenarios requiring data validation and processing.