Goose Perception: Local Ambient AI That Watches, Listens, and Learns

Goose Perception is an open-source macOS menu bar application that acts as a personal ambient intelligence assistant. Unlike cloud-based assistants that send your data to remote servers, Goose Perception runs entirely on your Mac: it captures your screen, voice, and face continuously, analyzes everything with on-device large language models (no cloud, no tracking, no API keys), and surfaces insights about your work patterns, collaborators, interests, and wellbeing. The project is explicitly experimental and under active development—especially the automated actions layer—but the core capture pipeline and insight generation are already working. If you care about privacy and want to explore what ambient AI can do when it never leaves your machine, Goose Perception is a compelling reference implementation and a usable tool.

Why Ambient AI, and Why Local?

Most AI assistants today are reactive: you ask a question, they answer. Ambient AI flips that: it observes your context over time—what you're working on, who you talk to, how long you've been at the screen, whether you look stressed or focused—and proactively offers summaries, reminders, and wellness nudges without you having to remember to ask. The challenge is that continuous observation usually means sending sensitive data (screen content, voice, camera) to the cloud, which raises obvious privacy and compliance concerns.

Goose Perception solves this by keeping everything on-device. Screen frames are captured, OCR'd for text, and then discarded; no screenshots are stored. Voice is transcribed locally with WhisperKit. Face analysis runs via Apple's Vision framework. All of that raw data is written to a local SQLite database. A small on-device LLM (Qwen2.5-3B, quantized to 4-bit and run via MLX) then acts as the "brain": it reads from the database, extracts structured knowledge (projects, people, TODOs), generates insights (work summaries, patterns, wellness observations), and eventually drives actions (popups, reminders). No data ever leaves your machine except for the one-time download of the models. That makes Goose Perception a rare example of local-first, edge AI that is both ambitious in scope and serious about privacy.

What Goose Perception Actually Does

Goose Perception runs as a menu bar app. In the background it does three kinds of capture:

Screen Capture and OCR

Roughly every 20 seconds, the app captures the currently focused window (or screen region) using ScreenCaptureKit. The image is not stored. It is passed to Apple's Vision framework for optical character recognition (OCR). Only the extracted text is written to the database in a screen_captures table, along with a timestamp and optional app/window metadata. So over time you get a time-series of what was on your screen: code, documents, emails, browser content—all as text. This is the main signal for understanding what you're working on.

Voice Capture and Transcription

Audio from the microphone is captured and sent to WhisperKit, Apple's Swift-native port of OpenAI's Whisper model. The project uses a small English model (e.g. whisper-tiny.en, around 40MB) so that transcription can run on-device with low latency. Transcribed segments are stored in a voice_segments table. That gives the system a record of meetings, calls, and spoken notes—again, only as text, so the pipeline can reason about topics, collaborators, and commitments mentioned in speech.

Face Detection and Emotion

The camera is used only for presence and emotion signals, not for storing images. Apple's Vision framework provides face detection and high-level emotion or expression cues. These are written to a face_events table (e.g. timestamps, presence, and simple mood or stress indicators). Combined with screen and voice, this lets the system notice long stretches of focus, late-night sessions, or possible stress—which feeds into wellness and break reminders.

All of this runs continuously (when you enable it). You can turn screen, voice, or face capture on or off independently from the app's Services tab, so you stay in control of what is observed.

Architecture: Four Layers From Raw Data to Actions

The system is organized into four clear layers. Data flows one way: from capture into the database, then through refiners that build structured knowledge, then through insight generators that produce natural-language observations, and finally through action generators that decide when to show a popup or notification.

Layer 1: Data Capture (Continuous)

Screen (every ~20s) → OCR → text into screen_captures. Voice → WhisperKit → text into voice_segments. Face → Vision → events into face_events. Nothing is sent to the network. The database is the single source of truth for all raw observations.

Layer 2: Refiners (Periodic, e.g. Every 20 Minutes)

Refiners are LLM-powered modules that read from the raw tables and write structured knowledge into dedicated tables. They run on a schedule (e.g. every 20 minutes) so that the system doesn't overload the on-device model. Typical refiners include:

• ProjectsRefiner: Scans screen and voice text for project names, codebases, and document themes.
• CollaboratorsRefiner: Extracts people, teams, and contact names from emails, docs, and transcripts.
• InterestsRefiner: Identifies recurring topics and domains (e.g. "machine learning", "product roadmap").
• TodosRefiner: Finds task-like phrases ("TODO", "need to", "follow up with") and turns them into a structured todos table.

Each refiner uses the same on-device LLM (MLX-Swift-LM with Qwen2.5-3B-Instruct-4bit). The model is prompted with recent raw data and asked to output structured records that are then inserted or updated in the database. The result is a growing knowledge graph of projects, people, interests, and tasks—all derived locally from your own activity.

Layer 3: Insight Generators (Scheduled with Cooldowns)

Insight generators combine the refined knowledge with recent mood or activity signals and produce observations: short natural-language insights that describe what you've been doing, how you're progressing, or how you might be feeling. They run on a schedule and respect cooldowns so you aren't flooded. Examples:

• WorkSummary (e.g. 2h cooldown): Runs when there are enough screen captures and at least one project. Produces a brief summary of recent work.
• PatternDetector (e.g. 4h cooldown): Looks across projects and captures to spot context-switching or focus patterns.
• ProgressTracker (e.g. 1h cooldown): Tied to the todos table; comments on pending tasks and progress.
• Collaboration (e.g. 3h cooldown): Uses collaborators and voice activity to summarize who you've been working with.
• Wellness (e.g. 30m cooldown): Uses face events, time of day, and long sessions to detect overwork, stress, or late-night usage and generates wellness-related observations.

Insights are stored in an insights table and can be shown in the Insights tab of the dashboard. They are also the main input for the next layer.

Layer 4: Action Generators (When to Interrupt)

Action generators decide when to surface a popup or notification. They look at recent insights (and optionally raw mood) and apply policy: don't nag too often, respect cooldowns, and only act when the insight is actionable. Examples:

• WellnessAction (e.g. 45m cooldown): If there are multiple wellness-related insights (e.g. long session, stress signals), show a break or stretch reminder.
• ReminderAction (e.g. 60m cooldown): If TODOs have been pending for a long time (e.g. >2 hours), suggest a reminder.
• FocusAction (e.g. 45m cooldown): If insights mention context-switching or fragmentation, offer a focus nudge.
• LateNightAction (e.g. 60m cooldown): After a configured hour (e.g. 10pm) plus late-night insights, show a wind-down suggestion.

Triggered actions are stored in an actions table and shown as popups or in the Actions tab. The codebase is moving toward a more general automated actions system (e.g. running AppleScript or other local tools) inspired by the TinyAgent: Function Calling at the Edge paper, using a simplified "Tool Rag" approach—but that part is still work in progress.

Technology Stack in Detail

Requirements: macOS 14+ and Apple Silicon (M1/M2/M3). The 4-bit quantized LLM and WhisperKit are sized to run comfortably on these chips without cloud fallback.

Database Schema: What Gets Stored

The SQLite database holds both raw and refined data. Main tables include:

• screen_captures: OCR text, timestamps, optional app/window info.
• voice_segments: Transcribed speech and timestamps.
• face_events: Timestamps, presence, and emotion-related fields.
• projects, collaborators, interests, todos: Outputs of the refiners.
• insights: Generated observations with timestamps and optional categories.
• actions: Triggered popups/notifications (pending, completed, dismissed).
• app_usage, directory_activity: Aggregated app usage and recent file activity where implemented.

No images or audio files are stored—only text and structured records. That keeps the database small and avoids retaining the most sensitive raw media.

Dashboard Tabs: What You Can See and Control

The main window is organized into tabs:

• Services: Toggle screen, voice, and face capture on/off; view audio levels and current emotion if available.
• Knowledge: Mood summary, list of extracted projects, collaborators, interests, app usage, directory activity, and TODOs.
• Insights: Chronological list of generated insights (work summaries, patterns, progress, collaboration, wellness).
• Actions: Pending, completed, and dismissed action items (popups/notifications).
• Activity: Real-time event log of capture and processing activity.
• Captures: Historical screen captures (OCR text preview only, no images).
• LLM: Full LLM session history for refiners and generators (useful for debugging and tuning).

This gives you full transparency: you can see exactly what was captured, what was extracted, what insights were generated, and what actions were triggered.

Privacy and Security: 100% Local

Goose Perception is designed so that no sensitive data leaves your machine:

• All processing is on-device. No cloud APIs, no external LLM calls, no telemetry.
• No images are stored. Screenshots are processed for OCR and then discarded.
• No network use except for the one-time download of models (Whisper, Qwen).
• User control: You can disable screen, voice, or face capture at any time from the Services tab.

That makes it a strong fit for developers, researchers, and knowledge workers who want ambient assistance without sending screen content, voice, or camera data to third parties. It's also a useful reference architecture for building local-first, multi-modal agents that combine vision, speech, and LLMs on the edge.

Who It's For and When It Makes Sense

• Privacy-focused users who want a single machine to "understand" their work context and nudge them (breaks, reminders, summaries) without any cloud dependency.
• Developers and researchers interested in on-device LLMs, multi-modal pipelines (screen + voice + face), and agent-style systems (refiners, generators, actions) that they can read and extend.
• Product and UX people exploring how ambient AI can support focus, wellness, and task awareness without sacrificing privacy.

It is not a replacement for a full cloud assistant (e.g. broad knowledge, real-time web search). It is best thought of as a local ambient layer that learns from your own activity and acts only on that.

Limitations and Current Status

The project is experimental. The capture and insight pipeline is in place and working; the action system (especially automated actions beyond popups, e.g. AppleScript or "Tool Rag") is still being implemented and tested. So you can expect:

• Reliable: screen/voice/face capture, refiners, insight generators, and in-app display of knowledge and insights.
• In progress: robust, general-purpose action execution (e.g. running local scripts or tools) and fine-tuning of when to interrupt.

If you run into issues, the GitHub repo has an issue tracker and the codebase is open for inspection and contribution.

Quick Start and Development

Clone the repo and from the project root run:

just build   # Build the app
just run     # Run (builds if needed)
just clean   # Clean build artifacts

For development:

just watch   # Watch for changes and rebuild
just test    # Run tests
just lint    # Lint

You need just installed (or run the underlying Swift build commands from the justfile). The app will prompt for the usual macOS permissions (screen capture, microphone, camera) when you enable each capture type.

Summary

Goose Perception is an open-source, local-only ambient AI assistant for macOS. It captures screen (OCR), voice (WhisperKit), and face (Vision) on-device, stores only text and structured data in SQLite, and uses an on-device LLM (Qwen2.5-3B via MLX) to refine that into projects, collaborators, interests, and TODOs, then to generate insights (work summaries, patterns, wellness) and eventually actions (popups, reminders). Everything runs on your Mac; nothing is sent to the cloud. The project is experimental and the automated actions layer is still evolving, but the capture and insight pipeline already demonstrates a practical edge AI stack: multi-modal sensing, local LLM reasoning, and a clear path from raw data to knowledge to insights to actions. Whether you use it as a daily tool or as a reference for building your own private, on-device intelligence, Goose Perception is worth a close look. License: Apache-2.0.