MaskAgent Developer Documentation

Overview

MaskAgent is a privacy-first browser automation tool that runs entirely on your local machine. It combines a Chrome extension with Ollama to enable AI-driven browser interactions while automatically redacting sensitive information before it reaches the model.

What it Does

Given a natural language goal (e.g., "Fill out this contact form" or "Extract the main article text"), MaskAgent:

1. Analyzes the current page's DOM and visual state
2. Detects and redacts PII (emails, phone numbers, passwords, etc.)
3. Sends sanitized context to a local AI model via Ollama
4. Receives structured actions (CLICK, TYPE, SCROLL, SELECT, DONE)
5. Validates and executes actions on the page
6. Repeats until the goal is achieved

What it Doesn't Do

• No cloud dependencies — everything runs locally
• No data storage — page data is processed ephemerally
• No credential capture — passwords are redacted
• No cross-origin automation — works within a single tab

Quick Start

1. Clone the repository

git clone https://github.com/bhuvanesh-m-dev/maskagent.git
cd maskagent

2. Install Ollama & pull a model

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a supported model
ollama pull deepseek-coder

# Verify models
ollama list

3. Load the extension in Chrome

1. Open chrome://extensions/
2. Enable Developer Mode
3. Click Load unpacked
4. Select the maskagent/code_files/ directory

4. Run your first agent

1. Navigate to any webpage
2. Click the MaskAgent extension icon
3. Enter a goal (e.g., "Find and click the login button")
4. Click Run Agent

Requirements

Installation

Directory Structure

maskagent/
├── code_files/              # ← Extension source
│   ├── manifest.json        # Extension configuration
│   ├── background.js        # Service worker & AI orchestration
│   ├── content.js           # Page interaction & privacy
│   ├── popup.html           # UI
│   ├── popup.js             # UI controller
│   └── styles.css           # UI styling
├── img/                     # Assets
├── docs/                    # Documentation
└── README.md

Loading the Extension

1. Open Chrome and navigate to chrome://extensions/
2. Toggle Developer Mode on (top-right)
3. Click Load unpacked
4. Select the code_files/ directory inside the repository
5. Verify MaskAgent appears in your extensions list

Verifying Ollama Connection

# Check if Ollama is running
curl http://localhost:11434/api/tags

# If not, start it
ollama serve

# List installed models
ollama list

Architecture

MaskAgent is composed of three primary runtime components that communicate via Chrome's message-passing APIs.

Trust Boundaries

Privacy is enforced at multiple boundaries in the system:

Codebase Guide

manifest.json

Extension configuration defining permissions, entry points, and content script injection rules.

{
    "manifest_version": 3,
    "name": "MaskAgent",
    "permissions": ["activeTab", "storage", "scripting"],
    "host_permissions": [
        "http://localhost:11434/*",
        "https://*/",
        "http://*/"
    ],
    "background": { "service_worker": "background.js" },
    "action": { "default_popup": "popup.html" },
    "content_scripts": [
        {
            "matches": [""],
            "js": ["content.js"],
            "run_at": "document_idle"
        }
    ]
}

background.js

Purpose: Service worker orchestrating the agent loop, communicating with Ollama, and coordinating between popup and content script.

Key Functions

• runAgent(goal, model) — Initializes a new agent session
• agentLoop() — Main execution loop (get state → AI → execute → repeat)
• queryOllama(prompt) — Sends prompt to Ollama, parses JSON response
• validateAction(action) — Validates action schema and parameters
• executeAction(action) — Sends action to content script

State Management

let session = {
    id: string,
    tabId: number,
    goal: string,
    model: string,
    step: number,
    maxSteps: 10,
    history: [{ step, action, result }],
    status: 'idle' | 'running' | 'complete' | 'error'
};

content.js

Purpose: Runs in page context for DOM analysis, PII detection/redaction, screenshot capture, and action execution.

Key Functions

• getPageState(options) — Collects DOM, text, and optionally screenshot
• extractDOM() — Builds structured element list with bounding rects
• detectPII(dom) — Scans for sensitive patterns (email, phone, password, etc.)
• redactPII(dom) — Replaces sensitive values with ***REDACTED***
• captureScreenshot() — Captures canvas, redacts sensitive areas
• executeAction(action) — Executes CLICK, TYPE, SCROLL, SELECT, DONE

PII Detection Patterns

const patterns = {
    email: /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/,
    phone: /[\+\d]?[\( ]?\d{3}[\)\- ]?\d{3}[\- ]?\d{4}/,
    password: /password|passwd|pwd/i,
    creditCard: /\d{4}[\- ]?\d{4}[\- ]?\d{4}[\- ]?\d{4}/
};

popup.html & popup.js

Purpose: UI and controller for the extension popup. Handles goal input, model selection, and displays agent status/logs.

Agent Pipeline

Action Schema

The AI model must respond with a JSON object matching one of these schemas:

Privacy Architecture

MaskAgent's privacy model is built on the principle of data minimization — only the minimum necessary information is sent to the AI model.

DOM-Level Redaction

The content script scans the DOM and marks elements as sensitive based on:

• Field type: input[type="password"] → automatically sensitive
• Field name: name="email", name="phone", etc.
• Pattern matching: Email addresses, phone numbers, credit card numbers

Sensitive values are replaced with ***REDACTED*** before being sent to the background script.

Visual Redaction

When screenshots are captured (for vision models), the canvas is redacted by drawing black rectangles over sensitive element bounding boxes.

What the AI Sees

• DOM: Element tags, IDs, classes, attributes — but values are redacted
• Text: Page text with patterns replaced by [EMAIL], [PHONE], etc.
• Screenshot: Page image with sensitive areas masked in black
• Never: Passwords, credit card numbers, private addresses

Limitations

Ollama Integration

API Endpoint

MaskAgent communicates with Ollama via the local API endpoint: http://localhost:11434/api/generate

Request Format

POST /api/generate
{
    "model": "deepseek-coder",
    "prompt": "... sanitized context + goal ...",
    "stream": false
}

Supported Models

Model Selection

Models are selected via the popup dropdown. The extension does not automatically detect installed models — the user must choose one that is installed.

Verifying Models

# List installed models
ollama list

# Pull a model
ollama pull deepseek-coder

# Check model details
ollama show deepseek-coder

Models

MaskAgent currently supports three model types, each with different strengths and requirements.

DeepSeek-Coder (Recommended)

• Size: ~4.7GB
• Type: Text-only
• Strength: Excellent at code-like structured outputs
• Use: Most automation tasks, form filling, navigation

LLaVA

• Size: ~4.3GB
• Type: Vision + Text
• Strength: Understands visual layout and UI elements
• Use: Finding buttons by appearance, visual QA

Qwen2.5-VL

• Size: ~7GB
• Type: Vision + Text
• Strength: Alternative vision model with strong reasoning
• Use: Complex visual understanding tasks

Action Contract

The AI model must output actions in a specific JSON format. MaskAgent validates these actions before execution.

CLICK

{
    "type": "CLICK",
    "target": "#submit-button"
}

Clicks the element matching the target selector. Target can be ID (#id), class (.class), or any valid CSS selector.

TYPE

{
    "type": "TYPE",
    "target": "#email-input",
    "value": "test@example.com"
}

Types the given value into the target input/textarea. Triggers input and change events.

SCROLL

{
    "type": "SCROLL",
    "target": "down"
}

Scrolls the page. Valid targets: "up", "down", "top", "bottom".

SELECT

{
    "type": "SELECT",
    "target": "#country",
    "value": "US"
}

Selects an option in a <select> dropdown. Triggers a change event.

DONE

{
    "type": "DONE"
}

Signals that the task is complete. The agent loop will stop.

Testing

Local Test Page

Create a test HTML page with various form elements:

<!DOCTYPE html>
<html>
<body>
    <h2>Test Form</h2>
    <form>
        <label>Name: <input type="text" name="name" value="John Doe"></label><br>
        <label>Email: <input type="email" name="email" value="john@example.com"></label><br>
        <label>Phone: <input type="tel" name="phone" value="123-456-7890"></label><br>
        <label>Password: <input type="password" name="password" value="secret123"></label><br>
        <button type="submit">Submit</button>
    </form>
</body>
</html>

Privacy Tests

• Verify email addresses are redacted → [EMAIL]
• Verify phone numbers are redacted → [PHONE]
• Verify password fields are marked sensitive
• Verify screenshot redaction masks sensitive areas

Agent Tests

• Test CLICK on buttons and links
• Test TYPE into input fields
• Test SCROLL navigation
• Test SELECT dropdown options
• Test DONE completion detection

Negative Tests

• Missing element → agent should handle gracefully
• Invalid action → validation should reject
• Ollama unavailable → error handling
• Malformed model output → parse errors

Debugging

Extension Debugging

Popup Console:

1. Right-click the extension icon → Inspect Popup
2. View logs in the Console tab

Service Worker Console:

1. Go to chrome://extensions/
2. Find MaskAgent and click service worker (or background page)
3. View logs in the Console tab

Content Script Console:

1. Open the page's developer tools (F12 or Cmd+Opt+I)
2. Look for logs from content.js

Ollama Debugging

# Check Ollama status
ollama list

# View Ollama logs
ollama serve

# Test Ollama API directly
curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek-coder", "prompt": "Hello", "stream": false}'

Common Issues

Extending MaskAgent

Adding a New PII Detector

1. Open content.js
2. Locate the detectPII() function
3. Add your pattern to the patterns object
4. Update the redactPII() function if needed

// In content.js - detectPII()
const patterns = {
    email: /.../,
    phone: /.../,
    // Add your new pattern here
    customId: /CUST-\d{4}-\d{4}/
};

Adding a New Browser Action

1. Define the action schema in the prompt (background.js)
2. Update validation in validateAction() (background.js)
3. Implement execution in executeAction() (content.js)
4. Add tests (see Testing)
5. Update documentation

// In content.js - executeAction()
case 'NEW_ACTION':
    // Implement your logic
    // Return { success: true, action: 'new_action' }
    break;

Adding a New AI Model

1. Add the model to the <select> in popup.html
2. The model must support the same API contract (JSON response)
3. Test with the new model

Contributing

Development Workflow

1. Fork the repository on GitHub
2. Clone your fork locally
3. Create a branch for your change
4. Make changes following the code style
5. Test locally (see Testing)
6. Verify privacy — no sensitive data in logs
7. Update documentation if needed
8. Commit with clear messages
9. Push to your fork
10. Open a Pull Request

Pull Request Requirements

• Explain what changed and why
• Describe how you tested it
• Confirm no privacy regression occurred
• Update documentation if you changed behavior

Areas for Contribution

• Privacy: Better PII detection, visual redaction, OCR
• Agent Intelligence: Planning, better validation, task completion
• Security: Prompt injection, action permissions, auditing
• Performance: Faster inference, DOM extraction, context compression
• Browser Compatibility: More browsers, better support

Security Model

Local Inference

All AI processing runs locally via Ollama. No data leaves your machine. The extension only communicates with localhost:11434.

Permission Model

• activeTab — Only active tab access
• storage — Local settings only
• scripting — Required for content script injection
• host_permissions — For Ollama and web pages

Action Validation

• All AI-generated actions are validated before execution
• Invalid actions are rejected with errors
• Actions are limited to safe operations (no destructive commands)

Known Risks

Limitations