Ponkotsu LLM

● Live

A lightweight language model running on a Raspberry Pi. It's no good at hard problems, but for small talk and a bit of writing help, it does its honest best. Responses stream back token by token.

Demo

chat.demo

⌘/Ctrl + Enter0 / 2000

Ponkotsu protection: input caps at 2000 characters, replies get cut at ~512 tokens.

Use cases

▸ Embed it as a one-shot FAQ "concierge" that answers common questions in your app or site.
▸ Rephrase or summarize a short bit of text on the spot.

How to use

OpenAI-compatible. Issue an API key (free), then use the official openai package and just point the baseURL at llm.ponkotsu-lab.net.

Install

npm install openai

Try it

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm.ponkotsu-lab.net",
  apiKey: "your issued key",
});

const stream = await client.chat.completions.create({
  model: "ponkotsu",
  messages: [{ role: "user", content: "Write a short story" }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); // print each token as it arrives
}

Run it and the text streams into your terminal piece by piece.

Notes

The model is ponkotsu (a lightweight gemma-based model).
It speaks the OpenAI Chat Completions dialect, so compatible clients and libraries work as-is.
In the browser, use new OpenAI({ ..., dangerouslyAllowBrowser: true }).

Limitations (the ponkotsu bits)

Free keys are capped at 3 requests/min, and issuance at 2 keys per day.
Being underpowered, long text and complex reasoning are not its strength.
Under load you may be rate-limited and put in a queue.