VALSEA Classroom Copilot

Overview

In many Vietnamese university lectures, instructors explain in Vietnamese and drop English technical terms mid-sentence: "the middleware sẽ chạy trước khi request tới controller". Students take notes in one language and search in the other, and generic speech tools tend to stumble exactly where the terminology is densest.

Classroom Copilot listens to the lecture in real time and turns the stream into things a student can actually use: a live transcript, a short Vietnamese summary, key terms, a plain-language explanation, an English recap and a quiz. It was built solo for the VALSEA EdTech Hack Sprint – Real-Time Classroom Assist track.

The idea in one line

VALSEA handles the hard voice-and-language layer for Vietnamese–English code-switching; Classroom Copilot converts that stream into notes, explanations and quizzes students can use immediately.

Architecture

The browser never sees a secret. All VALSEA and LLM keys live in the Fastify gateway, which also normalizes the transcript events before the frontend renders them.

What it does

• Live transcript with partial and final ASR events.
• Live assist (optional): a micro-summary and explanation per segment while the lecture is still going.
• Learning output: Vietnamese summary, key terms with context, a simple Vietnamese explanation, an English recap, multiple-choice quiz questions and a list of possibly confusing points.
• Confusion marking: a student can flag "I didn't get that" with an optional note, and the flag is attached to the timeline.
• Session history and export to Markdown or JSON.
• Resilience: batch audio upload when the realtime connection is unstable, and a demo mode with built-in Vietnamese–English scripts so the app can be shown without a microphone.

Technical decisions

Six VALSEA endpoints, each with a safe fallback. The app uses realtime ASR, batch transcription, annotations, clarifications, translations and formatting. They run in parallel where possible, and each call fails independently: if the translation endpoint errors, the notes still render, and the error is recorded in the artifacts instead of breaking the session.

A small, explicit WebSocket contract. The client sends session.start, audio.chunk, learning.generate, confusion.mark and liveAssist.set; the gateway answers with transcript.partial, transcript.final, learning.output, assist.live and error (with a recoverability flag). Keeping the protocol this small made it easy to test with the demo mode and to reason about reconnects.

Audio processed in the browser. Float32 samples are downsampled to 16 kHz PCM16 mono on the client, which keeps the upstream bandwidth low and removes the need for server-side transcoding.

Optional persistence. Sessions live in memory by default; with Supabase credentials and the two SQL migrations, history and confusion events become durable. That keeps local setup to two environment files.

Split deployment. The Next.js client deploys to Vercel; the gateway runs on a long-lived host (Render, Railway or Fly) because WebSockets need a persistent process. Production uses wss://.

Limitations I kept honest

It needs a VALSEA API key for ASR and enrichment and an OpenAI-compatible LLM for the structured notes. Realtime quality depends on the network, which is why batch upload and demo mode exist. And generated quizzes are only as good as the transcript they are built from.

What I took away

Hackathon constraints are a good filter: the features that survived were the ones that make the live loop trustworthy (partial transcripts, fallbacks, a demo mode), not the ones that sound impressive. Treating every external API as something that can fail independently turned out to be the most valuable design rule.