Show HN: Clippy – screen-aware voice AI in the browser

Sobre

A friend and I built a browser prototype that answers questions about whatever’s on your screen using getDisplayMedia, client-side wake-word detection, and server-side multimodal inference.<p>Hard parts:<p>– Getting the model to point to specific UI elements<p>– Keeping it coherent across multi-step workflows (“Help me create a sword in Tinkercad”)<p>– Preventing the infinite mirror effect and confusion between window vs full-screen sharing<p>– Keeping voice → screenshot → inference → voice late