🎙 VoiceMail — Documentation

How it works

Every voice command goes through a 5-step pipeline on the server. The browser receives live updates via Server-Sent Events (SSE) so the UI updates as each step completes rather than waiting for everything at once.

Cost per command

Approximate OpenAI API cost per voice command (using OpenAI TTS):

Voice commands

Speak naturally — the intent parser understands full sentences, not just keywords. Examples:

Reading emails

Time filters

Navigating

Replying & composing

Filters & toggles

The filter bar at the top of the app provides quick controls that apply to every voice command.

TTS engines

Change the default engine in .env:

TTS_ENGINE=openai   # or: pocket

Override per-session using the toggle button in the app, or per-request via the X-TTS-Engine header.

Reconnecting accounts

If any account shows an amber Reconnect button, its token has expired or was never saved. Gmail and Outlook use different flows.

Gmail — web browser

Tap Reconnect next to the disconnected Gmail account. A new tab opens, redirects to Google sign-in, and returns automatically when done.

Gmail — iOS app

The same button works from the iOS app, but the reconnect URL is automatically routed through ASWebAuthenticationSession instead of WKWebView. See iOS OAuth.

Outlook — device code flow

Tap Reconnect on the Work Outlook account. A page opens showing a short code and a Microsoft URL. Open the URL, enter the code, sign in — the page auto-detects completion and shows a confirmation.

POST /api/auth/register

POST Public — no auth

Create a new user account. Returns a JWT on success.

Request

POST /api/auth/register
Content-Type: application/json

{ "email": "you@example.com", "password": "yourpassword" }

Response — success (200)

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "user": { "id": 1, "email": "you@example.com", "plan": "free" }
}

Response — email already registered (400)

{ "detail": "Email already registered." }

POST /api/auth/login

POST Public — no auth

Sign in to an existing account. Returns a JWT.

Request

POST /api/auth/login
Content-Type: application/json

{ "email": "you@example.com", "password": "yourpassword" }

Response — success (200)

{ "token": "eyJhbGci...", "user": { "id": 1, "email": "you@example.com", "plan": "free" } }

Response — wrong credentials (401)

{ "detail": "Invalid email or password." }

GET /api/auth/me

GET Requires Authorization header

Verify a stored token and fetch the current user's profile.

Request

GET /api/auth/me
Authorization: Bearer eyJhbGci...

Response

{ "id": 1, "email": "you@example.com", "plan": "free", "is_active": true }

GET /api/health

GET Public — no auth

Health check. Use this to verify the server is running.

curl https://gotenor.totesoft.com/api/health

Response

{ "status": "ok", "auth": "jwt" }

GET /api/status

GET Requires Authorization header

Returns the connection status of all configured email accounts.

Request

GET /api/status
Authorization: Bearer eyJhbGci...

Response

{
  "status": "ok",
  "accounts": [
    { "label": "Rojavaasam Gmail",   "connected": true,  "type": "gmail" },
    { "label": "Ajishra Email",      "connected": false, "type": "gmail" },
    { "label": "Work Outlook",       "connected": true,  "type": "outlook" }
  ],
  "total_accounts": 3,
  "all_connected": false
}

Accounts with "connected": false are shown with a Reconnect button in the app UI.

GET /api/email/accounts

GET Requires Authorization header

Extended account list including credential type and re-auth capability.

Response

{
  "accounts": [
    {
      "label": "Ajishra Email",
      "connected": false,
      "type": "gmail",
      "can_reauth": true,
      "credential_type": "web"
    }
  ]
}

GET /api/email/gmail/reauth

GET Public — browser redirect

Starts the Gmail OAuth flow for a given account. Redirects the browser to Google sign-in. On success, the token is saved server-side.

Query parameters

GET /api/email/outlook/reauth

GET Public — browser page

Starts the Outlook re-auth using MSAL device code flow. Returns an HTML page showing a short code and the Microsoft sign-in URL. A background task polls for completion; when the user signs in, the page updates automatically.

Flow

GET /api/email/outlook/poll

GET Public — polled by device code page

Returns the current state of a pending Outlook device code session. Called automatically by the page returned from /outlook/reauth.

Query parameter

Responses

{ "status": "pending" }                             // still waiting
{ "status": "connected", "label": "Work Outlook" }   // success
{ "status": "error",     "detail": "expired_token: ..." } // failed

POST /api/voice

POST Requires Authorization header

The core endpoint. Accepts an audio file (multipart/form-data) and streams Server-Sent Events (SSE) back as each pipeline step completes.

Request headers

Request body

POST /api/voice
Content-Type: multipart/form-data
Authorization: Bearer eyJhbGci...
X-TTS-Engine: pocket

audio=@recording.webm

SSE response stream

The response is a stream of text/event-stream events. Parse each line starting with data: as JSON.

Example SSE stream

data: {"event":"transcript","text":"check my Ajishra email this week"}

data: {"event":"processing","text":"Checking your emails in Ajishra (this week)..."}

data: {"event":"response","text":"You have 2 emails this week in Ajishra..."}

data: {"event":"audio","data":"SUQzBAAA...","engine":"openai"}

data: {"event":"done"}

Consuming SSE from any frontend

const res = await fetch("https://gotenor.totesoft.com/api/voice", {
  method: "POST",
  headers: {
    "Authorization": "Bearer " + token,
    "X-TTS-Engine":  "openai",
  },
  body: formData,   // FormData with field name "audio"
});

const reader  = res.body.getReader();
const decoder = new TextDecoder();
let   buffer  = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop();
  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const event = JSON.parse(line.slice(6));
    // handle: transcript, processing, response, audio, suggestions, error, done
  }
}

Environment variables

Building an external frontend

Any app (React, Flutter, Swift, Android) can connect to the VoiceMail API. Follow this flow:

Step 1 — Register or log in

POST /api/auth/login
{ "email": "you@example.com", "password": "yourpassword" }
→ { "token": "eyJhbGci..." }

Store the JWT in secure storage. Attach it as Authorization: Bearer <token> on every request.

Step 2 — Check account status

GET /api/status
Authorization: Bearer eyJhbGci...

Step 3 — Send voice commands

Record audio → POST to /api/voice → parse the SSE stream as shown in the API reference above.

CORS

All origins are allowed (Access-Control-Allow-Origin: *). No additional CORS setup needed.

iOS TestFlight wrapper

The ios-wrapper/ directory in the repository contains a minimal native iOS app that loads the VoiceMail PWA in a full-screen WKWebView. It can be submitted to TestFlight for beta distribution.

Files

Xcode setup (5 minutes)

ASWebAuthenticationSession (iOS OAuth)

Google blocks OAuth sign-in inside WKWebView as a security policy. ASWebAuthenticationSession is Apple's solution — it opens OAuth flows in a separate sandboxed Safari process that Google trusts.

How it works in VoiceMail

The registered redirect URI in Google Cloud Console remains unchanged: https://gotenor.totesoft.com/api/email/gmail/callback. The voicemail:// scheme is only used for the final app callback after the token is saved — Google never sees it.