Opas: tokenisointiesimerkki korvattu oikealla kuvakaappauksella

- Staattinen tekstitokenisointiesimerkki korvattu kuvalla joka näyttää värikoodatut tokenit EN/FI-vertailussa - Markdown-renderöijään lisätty ![alt](src) kuvatuki - Kuva: static/images/tokenization-example.png Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wasm tokenize_js() exportti oppaan live-tokenizeria varten
2026-04-07 13:20:50 +03:00 · 2026-04-07 13:18:07 +03:00 · 2026-04-07 13:15:05 +03:00 · 2026-04-07 13:14:14 +03:00 · 2026-04-07 13:10:37 +03:00 · 2026-04-07 13:05:19 +03:00
6 changed files with 100 additions and 48 deletions
--- a/network-poc/docker-compose.yml
+++ b/network-poc/docker-compose.yml
@@ -36,12 +36,11 @@ services:
      dockerfile: Dockerfile.native-node
    container_name: kipina_native_node
    environment:
-      - HUB_URL=ws://agentic-poc:3000/ws
+      - HUB_URL=wss://kipina.studio/ws
      - OLLAMA_URL=http://ollama:11434
      - OLLAMA_MODEL=qwen2.5-coder:7b
      - ALLOCATED_GB=4
    depends_on:
-      - agentic-poc
      - ollama
    profiles:
      - native
--- a/network-poc/hub/Cargo.toml
+++ b/network-poc/hub/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "hub"
-version = "0.2.2"
+version = "0.2.3"
 edition = "2024"

 [dependencies]
--- a/network-poc/node/src/lib.rs
+++ b/network-poc/node/src/lib.rs
@@ -118,6 +118,27 @@ async fn run_ai_tensor_inference(difficulty: usize) -> String {
    format!("PoC {} Matmul ({}x{}) >> {}", backend_name, active_workload_size, active_workload_size, result)
 }

+/// JS-exportti: tokenisoi tekstin ja palauttaa JSON-merkkijonon
+/// Tokenizer ladataan IndexedDB:stä (täytyy olla ladattu aiemmin)
+#[wasm_bindgen]
+pub async fn tokenize_js(text: String) -> Result<String, JsValue> {
+    let cached_tok = storage::load_from_idb("tokenizer.json").await.unwrap_or(None);
+    let Some(bytes) = cached_tok else {
+        // Yritetään ladata verkosta
+        let resp = reqwest::get("https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B/resolve/main/tokenizer.json").await
+            .map_err(|e| JsValue::from_str(&format!("Tokenizer-lataus epäonnistui: {}", e)))?;
+        let bytes = resp.bytes().await
+            .map_err(|e| JsValue::from_str(&format!("Tokenizer-lataus epäonnistui: {}", e)))?;
+        let _ = storage::save_to_idb("tokenizer.json", &bytes).await;
+        let tokenizer = tokenizers::Tokenizer::from_bytes(&bytes)
+            .map_err(|e| JsValue::from_str(&format!("Tokenizer-parsinta: {}", e)))?;
+        return Ok(tokenize_text(&tokenizer, &text).to_string());
+    };
+    let tokenizer = tokenizers::Tokenizer::from_bytes(&bytes)
+        .map_err(|e| JsValue::from_str(&format!("Tokenizer-parsinta: {}", e)))?;
+    Ok(tokenize_text(&tokenizer, &text).to_string())
+}
+
 /// Tokenisoi yhden tekstin ja palauttaa metriikat
 fn tokenize_text(tokenizer: &tokenizers::Tokenizer, text: &str) -> serde_json::Value {
    let char_count = text.chars().count();
--- a/network-poc/static/GUIDE.md
+++ b/network-poc/static/GUIDE.md
@@ -36,29 +36,13 @@ sanan osa, kokonainen sana tai välilyönti. Tokenisaatio tehdään
 BPE-algoritmilla (Byte Pair Encoding) joka oppii yleisimmät
 merkkijonot harjoitusdatasta.

-### Esimerkki: koodi
-
-```
-"print('Hello')" → [print] [(' ] [Hello] [')]  = 4 tokenia
-"tulosta('Hei')" → [tul] [osta] [(' ] [He] [i] [')]  = 6 tokenia
-```
-
-Koodi tokenisoidaan tehokkaasti koska `print`, `def`, `return` yms.
-ovat kokonaisia tokeneita. Suomenkielinen `tulosta` joudutaan pilkkomaan
-osiin koska se ei esiinny harjoitusdatassa kokonaisena.
-
 ### Esimerkki: suomi vs. englanti

-Sama lause kahdella kielellä Qwen2.5-Coder -tokenisaattorilla:
+Alla oikea tokenisointitulos Qwen2.5-Coder-tokenisaattorilla. Jokainen
+värikoodattu lohko on yksi tokeni — huomaa miten suomi vaatii enemmän
+tokeneita saman merkityksen välittämiseen:

-| | Teksti | Tokenit | Määrä | Merkkejä/token |
-|---|---|---|---|---|
-| EN | The cat sat on the mat | [The] [ cat] [ sat] [ on] [ the] [ mat] | **6** | 3.7 |
-| FI | Kissa istui matolla | [K] [issa] [ ist] [ui] [ mat] [olla] | **6** | 3.2 |
-| EN | Distributed computing in the browser | [Dist] [ributed] [ computing] [ in] [ the] [ browser] | **6** | 6.0 |
-| FI | Hajautettu laskenta selaimessa | [H] [aj] [au] [tettu] [ las] [kenta] [ sel] [aim] [essa] | **9** | 3.3 |
-| EN | Write a function that sorts a list | [Write] [ a] [ function] [ that] [ sorts] [ a] [ list] | **7** | 5.0 |
-| FI | Kirjoita funktio joka lajittelee listan | [K] [irj] [oita] [ funkt] [io] [ joka] [ laj] [ittel] [ee] [ listan] | **10** | 4.0 |
+![Tokenisointivertailu EN/FI](/images/tokenization-example.png)

 **Huomaa miten:**
 - Englannin yleiset sanat (`the`, `in`, `a`, `function`) ovat kokonaisia tokeneita
--- a/network-poc/static/images/tokenization-example.png
+++ b/network-poc/static/images/tokenization-example.png
--- a/network-poc/static/index.html
+++ b/network-poc/static/index.html
@@ -463,8 +463,8 @@
        }
        
        .avatar-card img {
-            width: 48px;
-            height: 48px;
+            width: 50px;
+            height: 50px;
            border-radius: 12px;
            margin-bottom: 4px;
            border: 2px solid rgba(240, 246, 252, 0.1);
@@ -2840,26 +2840,62 @@ ${fixableFiles}`;
            const agentNames = { manager: 'Manageri', coder: 'Koodari', tester: 'DevOps', qa: 'QA', data: 'Data' };
            const agentColors = { manager: '#d29922', coder: '#3fb950', tester: '#58a6ff', qa: '#a371f7', data: '#d2a8ff' };

+            // Syntaksikorostus: kevyt regex-pohjainen highlighter
+            function highlightCode(code, filename) {
+                let h = code.replace(/&/g,'&amp;').replace(/</g,'&lt;');
+                if (filename.endsWith('.py') || filename.endsWith('.toml') || filename.endsWith('.yml') || filename.endsWith('.yaml')) {
+                    // Kommentit
+                    h = h.replace(/(#[^\n]*)/g, '<span style="color:#8b949e;font-style:italic">$1</span>');
+                    // Stringit (kolmois- ja yksittäiset)
+                    h = h.replace(/("""[\s\S]*?"""|'''[\s\S]*?''')/g, '<span style="color:#a5d6ff">$1</span>');
+                    h = h.replace(/((?<![\\])(?:"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'))/g, '<span style="color:#a5d6ff">$1</span>');
+                    // Avainsanat
+                    h = h.replace(/\b(def|class|import|from|return|if|elif|else|for|in|while|try|except|finally|with|as|raise|yield|async|await|True|False|None|not|and|or|is|lambda)\b/g, '<span style="color:#ff7b72">$1</span>');
+                    // Dekoraattorit
+                    h = h.replace(/^(\s*@\w+)/gm, '<span style="color:#d2a8ff">$1</span>');
+                    // Numerot
+                    h = h.replace(/\b(\d+\.?\d*)\b/g, '<span style="color:#79c0ff">$1</span>');
+                } else if (filename.endsWith('.html')) {
+                    h = h.replace(/(&lt;\/?[\w-]+)/g, '<span style="color:#7ee787">$1</span>');
+                    h = h.replace(/([\w-]+)=/g, '<span style="color:#79c0ff">$1</span>=');
+                    h = h.replace(/((?<!=)"[^"]*")/g, '<span style="color:#a5d6ff">$1</span>');
+                } else if (filename === 'Dockerfile') {
+                    h = h.replace(/^(FROM|RUN|COPY|WORKDIR|EXPOSE|CMD|ENV|ARG|ENTRYPOINT|ADD|VOLUME|LABEL|USER)/gm, '<span style="color:#ff7b72">$1</span>');
+                    h = h.replace(/(#[^\n]*)/g, '<span style="color:#8b949e;font-style:italic">$1</span>');
+                    h = h.replace(/((?<!=)"[^"]*")/g, '<span style="color:#a5d6ff">$1</span>');
+                }
+                return h;
+            }
+
            const stepsHtml = steps.map((s, i) => {
                const color = agentColors[s.agent] || '#8b949e';
                const icon = s.status === 'done' ? '✓' : '◷';
                const outputPreview = (s.output || '').substring(0, 500);
+                const highlighted = outputPreview ? highlightCode(outputPreview, s.label) : '';
                return `
-                    <div style="margin-bottom:16px;border:1px solid #30363d;border-radius:6px;overflow:hidden">
-                        <div style="background:#161b22;padding:8px 12px;display:flex;justify-content:space-between;align-items:center">
-                            <span><span style="color:${s.status === 'done' ? '#3fb950' : '#d29922'}">${icon}</span> <strong style="color:${color}">${agentNames[s.agent] || s.agent}</strong> — ${s.label}</span>
-                            <span style="color:#8b949e;font-size:12px">Vaihe ${i + 1}</span>
+                    <div style="margin-bottom:12px;border:1px solid #30363d;border-radius:8px;overflow:hidden">
+                        <div style="background:#161b22;padding:10px 14px;display:flex;justify-content:space-between;align-items:center;border-bottom:1px solid #21262d">
+                            <span><span style="color:${s.status === 'done' ? '#3fb950' : '#d29922'};font-size:14px">${icon}</span> <strong style="color:${color}">${agentNames[s.agent] || s.agent}</strong> <span style="color:#8b949e">—</span> ${s.label}</span>
+                            <span style="color:#484f58;font-size:11px;background:#0d1117;padding:2px 8px;border-radius:10px">Vaihe ${i + 1}</span>
                        </div>
-                        ${s.input ? `<details><summary style="padding:6px 12px;color:#8b949e;font-size:12px;cursor:pointer">Prompti</summary><pre style="margin:0;padding:8px 12px;background:#010409;font-size:11px;overflow-x:auto;white-space:pre-wrap;color:#8b949e">${s.input.replace(/</g,'&lt;').substring(0, 1000)}</pre></details>` : ''}
-                        ${outputPreview ? `<pre style="margin:0;padding:8px 12px;background:#0d1117;font-size:12px;overflow-x:auto;white-space:pre-wrap;color:#c9d1d9">${outputPreview.replace(/</g,'&lt;')}</pre>` : ''}
+                        ${s.input ? `<details><summary style="padding:6px 14px;color:#8b949e;font-size:12px;cursor:pointer;border-bottom:1px solid #21262d">▸ Prompti</summary><pre style="margin:0;padding:10px 14px;background:#010409;font-size:11px;overflow-x:auto;white-space:pre-wrap;color:#8b949e;line-height:1.5">${s.input.replace(/</g,'&lt;').substring(0, 1000)}</pre></details>` : ''}
+                        ${highlighted ? `<pre style="margin:0;padding:10px 14px;background:#0d1117;font-size:12px;overflow-x:auto;white-space:pre-wrap;color:#c9d1d9;line-height:1.6">${highlighted}</pre>` : ''}
                    </div>`;
            }).join('');

-            const filesHtml = fileEntries.map(([name, content]) => `
-                <div style="margin-bottom:16px;border:1px solid #30363d;border-radius:6px;overflow:hidden">
-                    <div style="background:#161b22;padding:8px 12px;font-weight:600;color:#58a6ff">${name}</div>
-                    <pre style="margin:0;padding:12px;background:#010409;font-size:12px;overflow-x:auto;white-space:pre-wrap;color:#c9d1d9">${(content || '').replace(/</g,'&lt;')}</pre>
-                </div>`).join('');
+            const filesHtml = fileEntries.map(([name, content]) => {
+                const ext = name.split('.').pop();
+                const langLabel = {py:'Python',html:'HTML',toml:'TOML',yml:'YAML',yaml:'YAML',md:'Markdown'}[ext] || ext.toUpperCase();
+                const lines = (content || '').split('\\n').length;
+                return `
+                <div style="margin-bottom:12px;border:1px solid #30363d;border-radius:8px;overflow:hidden">
+                    <div style="background:#161b22;padding:8px 14px;display:flex;justify-content:space-between;align-items:center;border-bottom:1px solid #21262d">
+                        <span style="font-weight:600;color:#58a6ff">${name}</span>
+                        <span style="color:#484f58;font-size:11px">${langLabel} · ${lines} riviä</span>
+                    </div>
+                    <pre style="margin:0;padding:12px 14px;background:#010409;font-size:12px;overflow-x:auto;white-space:pre-wrap;color:#c9d1d9;line-height:1.6">${highlightCode(content || '', name)}</pre>
+                </div>`;
+            }).join('');

            const staticHtml = (staticIssues || []).length > 0
                ? `<div style="margin-bottom:16px;padding:12px;background:#1c1206;border:1px solid #d29922;border-radius:6px">
@@ -2876,14 +2912,18 @@ ${fixableFiles}`;
 <title>Kipinä Raportti — ${task}</title>
 <style>
  * { box-sizing: border-box; margin: 0; padding: 0; }
-  body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; background: #0d1117; color: #c9d1d9; padding: 20px; max-width: 900px; margin: 0 auto; }
-  h1 { color: #58a6ff; margin-bottom: 4px; }
-  h2 { color: #c9d1d9; margin: 24px 0 12px; border-bottom: 1px solid #30363d; padding-bottom: 6px; }
+  body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, sans-serif; background: #0d1117; color: #c9d1d9; padding: 24px; max-width: 960px; margin: 0 auto; line-height: 1.5; }
+  h1 { color: #f0f6fc; margin-bottom: 4px; font-size: 24px; }
+  h2 { color: #c9d1d9; margin: 28px 0 14px; border-bottom: 1px solid #30363d; padding-bottom: 8px; font-size: 18px; }
  h3 { color: #8b949e; margin: 16px 0 8px; }
-  pre { font-family: 'Courier New', monospace; }
+  pre { font-family: ui-monospace, "Cascadia Code", "SF Mono", Menlo, monospace; tab-size: 4; }
  a { color: #58a6ff; }
-  details summary { list-style: none; }
+  details summary { list-style: none; cursor: pointer; }
  details summary::-webkit-details-marker { display: none; }
+  details[open] summary { color: #58a6ff; }
+  .sl-badge { display: inline-block; border: 1px solid; border-radius: 6px; padding: 3px 10px; margin: 2px; font-size: 11px; cursor: help; white-space: nowrap; position: relative; transition: transform 0.15s; }
+  .sl-badge:hover { transform: translateY(-1px); filter: brightness(1.3); }
+  .sl-badge[data-tip]:hover::after { content: attr(data-tip); position: absolute; bottom: calc(100% + 6px); left: 50%; transform: translateX(-50%); background: #1c2028; border: 1px solid #30363d; border-radius: 6px; padding: 8px 12px; font-size: 11px; color: #c9d1d9; white-space: pre-wrap; max-width: 350px; min-width: 180px; z-index: 10; box-shadow: 0 4px 12px rgba(0,0,0,0.5); pointer-events: none; line-height: 1.5; }
 </style>
 </head>
 <body>
@@ -2903,7 +2943,7 @@ ${staticHtml}
 ${filesHtml}

 <hr style="border-color:#30363d;margin:24px 0">
-<p style="color:#8b949e;font-size:12px">Generoitu Kipinä Agentic Playground v0.2.2 — <a href="https://kipina.studio">kipina.studio</a></p>
+<p style="color:#8b949e;font-size:12px">Generoitu Kipinä Agentic Playground v0.2.3 — <a href="https://kipina.studio">kipina.studio</a></p>
 </body></html>`;
        }

@@ -2932,16 +2972,23 @@ ${filesHtml}
                for (var bi = 0; bi < aSteps.length; bi++) {
                    var st = aSteps[bi];
                    var desc = stepDescs[st.label] || st.label;
-                    var outPrev = (st.output || '').substring(0, 100).replace(/</g, '&lt;').replace(/\n/g, ' ');
-                    var tip = desc + (outPrev ? ' — ' + outPrev : '');
+                    var outPrev = (st.output || '').substring(0, 150).replace(/</g, '&lt;').replace(/"/g, '&quot;');
+                    // Rivitys tooltipiin: jokainen rivi omalle rivilleen
+                    var tipLines = [desc];
+                    if (outPrev) {
+                        tipLines.push('');
+                        var outLines = outPrev.split(/\n/).slice(0, 6);
+                        for (var li = 0; li < outLines.length; li++) tipLines.push(outLines[li]);
+                        if ((st.output || '').split(/\n/).length > 6) tipLines.push('...');
+                    }
                    var icon = st.status === 'done' ? '✓' : '◷';
-                    if (bi > 0) badges += '<span style="color:#30363d;margin:0 1px">→</span>';
-                    badges += '<span title="' + tip.replace(/"/g, '&quot;') + '" style="display:inline-block;background:' + bg + ';border:1px solid ' + color + ';border-radius:4px;padding:3px 8px;margin:2px;font-size:11px;color:' + color + ';cursor:help;white-space:nowrap">' + icon + ' ' + st.label.replace(/</g, '&lt;') + '</span>';
+                    if (bi > 0) badges += '<span style="color:#484f58;margin:0 2px">→</span>';
+                    badges += '<span class="sl-badge" style="background:' + bg + ';border-color:' + color + ';color:' + color + '" data-tip="' + tipLines.join('&#10;') + '">' + icon + ' ' + st.label.replace(/</g, '&lt;') + '</span>';
                }
-                rows += '<div style="display:flex;align-items:center;gap:8px;margin:4px 0"><span style="min-width:70px;font-weight:600;font-size:12px;color:' + color + '">' + label + '</span><div style="display:flex;flex-wrap:wrap;align-items:center">' + badges + '</div></div>';
+                rows += '<div style="display:flex;align-items:center;gap:10px;margin:5px 0;padding:4px 0;border-bottom:1px solid #21262d"><span style="min-width:70px;font-weight:600;font-size:12px;color:' + color + '">' + label + '</span><div style="display:flex;flex-wrap:wrap;align-items:center;gap:2px">' + badges + '</div></div>';
            }

-            return '<div style="background:#0d1117;border:1px solid #30363d;border-radius:6px;padding:12px">' + rows + '</div>';
+            return '<div style="background:#0d1117;border:1px solid #30363d;border-radius:8px;padding:14px 16px">' + rows + '</div>';
        }

        // Yksinkertainen pipeline (vanha: manageri → koodari → testaaja)
@@ -4517,6 +4564,7 @@ ${filesHtml}

            function inlineFormat(text) {
                return text
+                    .replace(/!\[([^\]]*)\]\(([^)]+)\)/g, '<img src="$2" alt="$1" style="max-width:100%;border-radius:8px;border:1px solid #30363d;margin:12px 0;display:block">')
                    .replace(/`([^`]+)`/g, '<code style="background:#161b22;padding:2px 6px;border-radius:3px;font-size:13px;color:#e6edf3">$1</code>')
                    .replace(/\*\*([^*]+)\*\*/g, '<strong style="color:#e6edf3">$1</strong>')
                    .replace(/\*([^*]+)\*/g, '<em>$1</em>');
Author	SHA1	Message	Date
jaakko	f75e7f07e9	Opas: tokenisointiesimerkki korvattu oikealla kuvakaappauksella - Staattinen tekstitokenisointiesimerkki korvattu kuvalla joka näyttää värikoodatut tokenit EN/FI-vertailussa - Markdown-renderöijään lisätty ![alt](src) kuvatuki - Kuva: static/images/tokenization-example.png Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-07 13:20:50 +03:00
jaakko	eb6f720fcc	Wasm tokenize_js() exportti oppaan live-tokenizeria varten Lisätty #[wasm_bindgen] tokenize_js(text) → JSON-funktio joka lataa tokenizerin IndexedDB:stä tai HuggingFacesta tarvittaessa. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-07 13:18:07 +03:00
jaakko	e25d0ea8f2	Versio 0.2.3 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-07 13:15:05 +03:00
jaakko	4d1e89da34	Avatar-kuvakkeet 48px → 50px (+5%) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-07 13:14:14 +03:00
jaakko	bf535b6256	Raportti: syntaksikorostus, CSS-tooltipsit rivityksellä, parannettu layout - Lisätty highlightCode() — regex-pohjainen korostus .py, .html, Dockerfile - Swimlane-badget: title-attr → CSS ::after tooltip (white-space:pre-wrap) - Tiedostokortit: kieli + rivimäärä headeriin - Yleinen ilme: pyöristetyt kulmat, monospace-fontti, parempi line-height - Pipeline-vaiheet: selkeämmät erottimet ja väripaletti Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-07 13:10:37 +03:00
jaakko	29e1c440c6	Native-node osoittaa nyt tuotantohubiin (kipina.studio) HUB_URL vaihdettu ws://agentic-poc:3000/ws → wss://kipina.studio/ws jotta lokaali GPU-solmu palvelee tuotantoympäristöä. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-07 13:05:19 +03:00