21 Commits

Author SHA1 Message Date
Jaakko Vanhala
ee048b0b68 kipina-node: automaattinen Ollama-instanssien haku + konttituki
Skripti skannaa localhost, 127.0.0.1, ollama, host.docker.internal
ja tarjoaa valikon jos useampi löytyy. Ei vaadi enää paikallista
ollama-binääriä — toimii myös Docker-konttia tai remote-instanssia
vasten. OLLAMA_URL välitetään Rust-binäärille.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 10:19:00 +03:00
Jaakko Vanhala
4e83569194 Konsoliloki näyttää mallin nimen: ✓ qwen2.5-coder:3b | 438 tok | 4952ms | 93.4 tok/s
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 10:01:32 +03:00
Jaakko Vanhala
f42b692eeb Lyhennetty konsolilogi: yksi rivi per pyyntö + yksi rivi per tulos
Ennen: koko prompti + vastaus logitettiin (satoja rivejä)
Jälkeen:
  → task_id:abc | 42r prompti | "Write ONLY models.py..."
  ✓ 128 tok | 3200ms | 40.0 tok/s | "from sqlalchemy import..."

llm_done-viestissä prompt lyhennetty viimeiseen riviin (ei koko kontekstia).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 10:00:39 +03:00
Jaakko Vanhala
f79bb16f3d kipina-node binäärijakelu: download-skripti + macOS ARM64 binääri
kipina.studio/kipina-node — shell-skripti joka:
1. Tunnistaa OS/arch (macOS ARM, Linux x86/ARM)
2. Tarkistaa Ollaman (asennettu? käynnissä?)
3. Lataa kielimallin automaattisesti
4. Lataa oikean binäärin kipina.studio/download/
5. Käynnistää noden → yhdistää hubiin

Käyttö: curl -sSL https://kipina.studio/kipina-node | bash
Tai:    curl -sSL https://kipina.studio/kipina-node -o kipina-node && chmod +x kipina-node && ./kipina-node

build-binaries.sh — kääntää binäärit kaikille alustoille (Docker).
macOS ARM64 binääri (4.9MB) valmis, Linux x86_64 build käynnissä.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 09:51:31 +03:00
Jaakko Vanhala
e81fc33faf Join-dialogi: kaksi selkeää vaihetta (Ollama + kipina-node binääri)
Vaihe 1: Asenna Ollama
  curl -fsSL https://ollama.ai/install.sh | sh
  (+ brew/Windows-vaihtoehdot)

Vaihe 2: Lataa ja käynnistä kipina-node
  curl -sSL https://kipina.studio/kipina-node -o kipina-node && chmod +x kipina-node && ./kipina-node

Ei vaadi Rustia — valmis binääri ladataan suoraan.
Molemmat komennot kopioitavissa yhdellä klikkauksella.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:56:00 +03:00
Jaakko Vanhala
433726c553 Palautettu docker-compose.prod.yml: vain Caddy + Hub (ei Ollamaa palvelimella)
Ollama ajetaan käyttäjien omilla koneilla join.sh:n kautta,
ei palvelimella. Selain-Wasm toimii fallbackina.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:53:06 +03:00
Jaakko Vanhala
dec2e24e2f "Liitä koneesi" -nappi + join.sh + Docker native-node
UI: status-palkissa vihreä "+ Liitä koneesi" -nappi joka avaa dialogin:
  curl -sSL https://kipina.studio/join.sh | bash

join.sh:
- Tarkistaa Ollaman → tarjoaa asennusta jos puuttuu
- Käynnistää Ollaman jos ei pyöri
- Lataa kielimallin (qwen2.5-coder:3b)
- Käynnistää native-noden → yhdistää wss://kipina.studio/ws

Docker: Dockerfile.native + docker-compose.prod.yml päivitetty
ollama + native-node -konteilla palvelinpuolelle.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:46:22 +03:00
Jaakko Vanhala
9058033669 Poistettu fonttiskaalaus (A-/A+) — ei vaikuttanut terminaaliin
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:33:20 +03:00
Jaakko Vanhala
8bd86e6325 Fonttikoon A-/A+ säädin: ±20% viidessä askeleessa
Oikeassa yläkulmassa A- ja A+ napit. Skaalaa 80-120%, tallennetaan localStorageen.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:31:33 +03:00
Jaakko Vanhala
c1133bb075 Terminaalin fontti 15→16px
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:27:31 +03:00
Jaakko Vanhala
6502d75efc Terminaalin syöttökenttä korostettu: sininen reunus, varjo, isompi fontti 16px
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:25:37 +03:00
Jaakko Vanhala
9f8b7fe920 UI-fonttikoot kasvatettu: body 16px, terminaali 15px, tabit 15px, status 14px
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:23:46 +03:00
Jaakko Vanhala
746bc20fcb Agenttikuvakkeet kasvatettu: 50→64px kuva, 72→90px kortti, isompi fontti
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:22:42 +03:00
Jaakko Vanhala
93f6baa0ea UI kasvatettu: container 1200→1600px, terminaali korkeampi, padding leveämpi
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:18:34 +03:00
Jaakko Vanhala
cc8e871735 deploy-fast.sh: luo hakemisto palvelimelle ennen rsync:iä
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:15:19 +03:00
Jaakko Vanhala
e90f3460c3 deploy-fast.sh: päivitä vain frontend ilman kontin uudelleenkäynnistystä
docker-compose.prod.yml: frontend/dist mountataan volumena (read-only).
Hub servaa tiedostot suoraan — rsync päivittää ne lennossa.

Kolme deploy-tasoa:
1. deploy-fast.sh — vain frontend (sekunteja, ei downtime)
2. deploy-light.sh — rsync + remote Docker build (minuutteja)
3. deploy.sh — lokaali build + image siirto (hidas mutta varma)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:12:10 +03:00
Jaakko Vanhala
4d74c38618 Dockerfile: poistettu turha COPY pkg (Astro kopioi public/:n automaattisesti)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:06:39 +03:00
Jaakko Vanhala
8a1b204179 v0.3.1: Avatarit WebP (18MB→256KB), PNG:t temp-kansioon
Kaikki avatar-viittaukset .png → .webp (200px, quality 80).
Alkuperäiset PNG:t siirretty temp/avatars-png/ (gitignored).
Hub-versio 0.3.0 → 0.3.1.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-11 08:06:02 +03:00
Jaakko Vanhala
b19f5a3518 deploy-light.sh: rsync + remote build (ei image-siirtoa)
Lähettää vain lähdekoodin rsync:llä (~2MB muuttuneet tiedostot),
palvelin buildaa Docker-imagen itse. Nopeampi kuin 80MB imagen siirto.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 21:56:01 +03:00
Jaakko Vanhala
38dc36e846 Dockerfile wasm-builder: lisätty dummy-cratet workspace-yhteensopivuuteen
cargo metadata vaatii kaikkien workspace-jäsenten Cargo.toml:n.
Lisätty hub/, native-node/, cli/ dummy-tiedostot wasm-builder-vaiheeseen.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 08:07:28 +03:00
Jaakko Vanhala
4fe6931b5f Revolutionized 2026-04-10 08:06:15 +03:00
151 changed files with 419 additions and 21945 deletions

3
.gitignore vendored
View File

@@ -37,3 +37,6 @@ Cargo.lock
# Ajonaikaiset tietokannat
*.db
# Wanha versio
temp/

View File

@@ -0,0 +1,21 @@
# Native-node: Rust + Ollama-client (ei GPU-tunnistusta)
FROM rust:slim AS builder
RUN apt-get update && apt-get install -y pkg-config libssl-dev && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY Cargo.toml Cargo.lock* ./
COPY native-node/Cargo.toml native-node/Cargo.toml
COPY native-node/src native-node/src
# Dummy-cratet workspace-yhteensopivuuteen
COPY hub/Cargo.toml hub/Cargo.toml
COPY node/Cargo.toml node/Cargo.toml
COPY cli/Cargo.toml cli/Cargo.toml
RUN mkdir -p hub/src node/src cli/src && touch hub/src/main.rs node/src/lib.rs cli/src/main.rs
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/app/target \
cargo build --release -p native-node --no-default-features \
&& cp /app/target/release/native-node /usr/local/bin/native-node
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=builder /usr/local/bin/native-node /usr/local/bin/native-node
CMD ["native-node"]

View File

@@ -3,10 +3,11 @@
# --- Vaihe 1: Frontend (Astro) ---
FROM node:22-slim AS frontend
WORKDIR /app/frontend
# Riippuvuudet ensin → cache-kerros (muuttuu harvoin)
COPY frontend/package.json frontend/package-lock.json* ./
RUN npm install --silent
# Lähdekoodi → muuttuu usein, mutta npm install on cachessa
COPY frontend/ .
COPY frontend/public/pkg public/pkg
RUN npm run build
# --- Vaihe 2: Wasm (wasm-pack) ---
@@ -17,6 +18,11 @@ WORKDIR /app
COPY Cargo.toml Cargo.lock* ./
COPY node/Cargo.toml node/Cargo.toml
COPY node/src node/src
# Dummy-cratet jotta workspace Cargo.toml on tyytyväinen
COPY hub/Cargo.toml hub/Cargo.toml
COPY native-node/Cargo.toml native-node/Cargo.toml
COPY cli/Cargo.toml cli/Cargo.toml
RUN mkdir -p hub/src native-node/src cli/src && touch hub/src/main.rs native-node/src/main.rs cli/src/main.rs
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/app/target \
cd node && wasm-pack build --target web --out-dir /app/wasm-pkg

38
network-poc/build-binaries.sh Executable file
View File

@@ -0,0 +1,38 @@
#!/bin/bash
# Käännä kipina-node binäärit kaikille alustoille
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
OUT="$SCRIPT_DIR/frontend/public/download"
mkdir -p "$OUT"
echo "=== Kipinä Node — Binary Build ==="
# macOS ARM (natiivi)
echo "[1/3] macOS ARM64..."
cd "$SCRIPT_DIR"
cargo build --release -p native-node --no-default-features 2>&1 | tail -1
cp target/release/native-node "$OUT/kipina-node-macos-arm64"
echo " $(ls -lh "$OUT/kipina-node-macos-arm64" | awk '{print $5}')"
# Linux x86_64 (Docker)
echo "[2/3] Linux x86_64..."
docker run --rm \
-v "$SCRIPT_DIR":/app -w /app \
--platform linux/amd64 \
rust:slim \
bash -c "apt-get update -qq && apt-get install -y -qq pkg-config libssl-dev >/dev/null 2>&1 && cargo build --release -p native-node --no-default-features 2>&1 | tail -1 && cp target/release/native-node /app/frontend/public/download/kipina-node-linux-x86_64"
echo " $(ls -lh "$OUT/kipina-node-linux-x86_64" | awk '{print $5}')"
# Linux ARM64 (Docker)
echo "[3/3] Linux ARM64..."
docker run --rm \
-v "$SCRIPT_DIR":/app -w /app \
--platform linux/arm64 \
rust:slim \
bash -c "apt-get update -qq && apt-get install -y -qq pkg-config libssl-dev >/dev/null 2>&1 && cargo build --release -p native-node --no-default-features 2>&1 | tail -1 && cp target/release/native-node /app/frontend/public/download/kipina-node-linux-arm64"
echo " $(ls -lh "$OUT/kipina-node-linux-arm64" | awk '{print $5}')"
echo ""
echo "=== Binäärit valmiina ==="
ls -lh "$OUT"/kipina-node-*

28
network-poc/deploy-fast.sh Executable file
View File

@@ -0,0 +1,28 @@
#!/bin/bash
# Nopea deploy: päivittää vain frontendin (ei kontin uudelleenkäynnistystä)
# Hub-binäärin päivitys: käytä deploy.sh tai deploy-light.sh
set -e
SERVER="ubuntu@86.50.252.98"
REMOTE_DIR="~/code/agentic-studio/network-poc"
SSH_OPTS="-o StrictHostKeyChecking=no"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
echo "=== Kipinä Studio — Frontend Deploy ==="
# 1. Buildaa frontend paikallisesti
echo "[1/2] Rakennetaan frontend..."
cd "$SCRIPT_DIR/frontend"
[ -d node_modules ] || npm install --silent
npm run build --silent 2>&1 | tail -1
# 2. Synkataan dist/ palvelimelle (vain muuttuneet tiedostot)
echo "[2/2] Synkataan dist/ → palvelin..."
ssh $SSH_OPTS $SERVER "mkdir -p $REMOTE_DIR/frontend/dist"
rsync -az --delete -e "ssh $SSH_OPTS" "$SCRIPT_DIR/frontend/dist/" "$SERVER:$REMOTE_DIR/frontend/dist/"
echo ""
echo "=== Valmis! Frontend päivitetty — ei uudelleenkäynnistystä ==="
echo " https://kipina.studio"
echo ""
echo "Huom: Jos Rust-koodi (hub/) muuttui, aja: ./deploy.sh"

33
network-poc/deploy-light.sh Executable file
View File

@@ -0,0 +1,33 @@
#!/bin/bash
# Kevyt deploy: lähetetään vain koodi, palvelin buildaa itse
set -e
SERVER="ubuntu@86.50.252.98"
REMOTE_DIR="~/code/agentic-studio/network-poc"
SSH_OPTS="-o StrictHostKeyChecking=no"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
echo "=== Kipinä Studio Deploy (remote build) ==="
# 1. Synkataan koodi palvelimelle (vain muuttuneet tiedostot)
echo "[1/3] Synkataan koodi..."
rsync -az --delete \
--exclude 'target/' \
--exclude 'node_modules/' \
--exclude 'dist/' \
--exclude '.astro/' \
--exclude 'temp/' \
--exclude '*.db' \
--exclude '.git/' \
"$SCRIPT_DIR/" "$SERVER:$REMOTE_DIR/"
# 2. Rakennetaan image palvelimella
echo "[2/3] Rakennetaan image palvelimella..."
ssh $SSH_OPTS $SERVER "cd $REMOTE_DIR && docker build -f Dockerfile.prod -t kipina-agentic:latest ."
# 3. Käynnistetään
echo "[3/3] Käynnistetään..."
ssh $SSH_OPTS $SERVER "cd $REMOTE_DIR && docker compose -f docker-compose.prod.yml down && docker compose -f docker-compose.prod.yml up -d"
echo "=== Valmis! https://kipina.studio ==="

View File

@@ -19,8 +19,12 @@ services:
restart: unless-stopped
environment:
- DATABASE_PATH=/data/nodes.db
- STATIC_DIR=/app/frontend/dist
- ADMIN_PASSWORD=${ADMIN_PASSWORD:-}
- NODE_API_KEY=${NODE_API_KEY:-}
volumes:
- hub_data:/data
- ./frontend/dist:/app/frontend/dist:ro
volumes:
caddy_data:

Binary file not shown.

Before

Width:  |  Height:  |  Size: 696 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 757 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 700 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.5 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 731 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 711 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.8 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 695 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 130 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 432 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 650 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.3 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 389 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.7 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 596 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.9 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 496 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.0 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 872 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 738 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 109 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 130 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.4 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 813 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 658 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 8.7 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 593 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 563 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.7 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 696 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 718 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.3 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 513 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 780 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 826 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

View File

@@ -0,0 +1,73 @@
#!/bin/bash
# Kipinä — liitä koneesi laskentaverkkoon
set -e
HUB_URL="${KIPINA_HUB:-wss://kipina.studio/ws}"
MODEL="${KIPINA_MODEL:-qwen2.5-coder:3b}"
echo ""
echo " ╔══════════════════════════════════════╗"
echo " ║ Kipinä Agentic Network — Node Join ║"
echo " ╚══════════════════════════════════════╝"
echo ""
# 1. Ollama
if command -v ollama &>/dev/null; then
echo " ✓ Ollama löytyi: $(ollama --version 2>/dev/null || echo 'asennettu')"
else
echo " Ollama ei ole asennettu."
echo ""
read -p " Asennetaanko Ollama? (k/e) " -n 1 -r; echo
if [[ $REPLY =~ ^[Kk]$ ]]; then
echo " Asennetaan Ollama..."
curl -fsSL https://ollama.ai/install.sh | sh
else
echo " Ollama vaaditaan laskentaan. Asenna: https://ollama.ai"
exit 1
fi
fi
# 2. Varmistetaan että Ollama on käynnissä
if ! curl -s http://localhost:11434/api/tags &>/dev/null; then
echo " Käynnistetään Ollama..."
ollama serve &>/dev/null &
sleep 3
if ! curl -s http://localhost:11434/api/tags &>/dev/null; then
echo " ✗ Ollama ei käynnistynyt. Aja: ollama serve"
exit 1
fi
fi
echo " ✓ Ollama käynnissä"
# 3. Malli
if ollama list 2>/dev/null | grep -q "$MODEL"; then
echo " ✓ Malli $MODEL ladattu"
else
echo " Ladataan malli $MODEL..."
ollama pull "$MODEL"
fi
# 4. Native-node
echo ""
echo " Yhdistetään hubiin: $HUB_URL"
echo " Malli: $MODEL"
echo " Ctrl+C pysäyttää"
echo ""
# Tarkistetaan onko native-node käännetty
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
NATIVE_BIN="$SCRIPT_DIR/target/release/native-node"
if [ -f "$NATIVE_BIN" ]; then
HUB_URL="$HUB_URL" OLLAMA_MODEL="$MODEL" "$NATIVE_BIN"
elif command -v cargo &>/dev/null && [ -f "$SCRIPT_DIR/native-node/Cargo.toml" ]; then
echo " Käännetään native-node..."
cd "$SCRIPT_DIR"
cargo build --release -p native-node --no-default-features 2>&1 | tail -1
HUB_URL="$HUB_URL" OLLAMA_MODEL="$MODEL" "$NATIVE_BIN"
else
echo " ✗ native-node binääriä ei löydy eikä Rust ole asennettu."
echo " Asenna Rust: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh"
echo " Tai lataa valmis binääri: https://kipina.studio/download"
exit 1
fi

View File

@@ -0,0 +1,121 @@
#!/bin/bash
# Kipinä Node — lataa oikea binääri ja käynnistä
set -e
BASE_URL="https://kipina.studio/download"
HUB_URL="${KIPINA_HUB:-wss://kipina.studio/ws}"
MODEL="${KIPINA_MODEL:-qwen2.5-coder:3b}"
OLLAMA_URL="${OLLAMA_URL:-http://localhost:11434}"
# Tunnista OS ja arkkitehtuuri
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m)
case "$OS-$ARCH" in
darwin-arm64) BINARY="kipina-node-macos-arm64" ;;
darwin-x86_64) BINARY="kipina-node-macos-arm64" ;; # Rosetta
linux-x86_64) BINARY="kipina-node-linux-x86_64" ;;
linux-aarch64) BINARY="kipina-node-linux-arm64" ;;
*) echo "Ei tuettu: $OS-$ARCH"; exit 1 ;;
esac
echo ""
echo " ╔══════════════════════════════════════╗"
echo " ║ Kipinä Agentic Node ║"
echo " ╚══════════════════════════════════════╝"
echo ""
echo " OS: $OS ($ARCH)"
echo ""
# Etsi Ollama-instanssit
CANDIDATES=(
"http://localhost:11434"
"http://127.0.0.1:11434"
"http://ollama:11434"
"http://host.docker.internal:11434"
)
# Lisää OLLAMA_URL listaan jos asetettu ja ei jo mukana
if [ -n "$OLLAMA_URL" ]; then
ALREADY=false
for c in "${CANDIDATES[@]}"; do
[ "$c" = "$OLLAMA_URL" ] && ALREADY=true
done
$ALREADY || CANDIDATES=("$OLLAMA_URL" "${CANDIDATES[@]}")
fi
echo " Etsitään Ollama-instansseja..."
FOUND=()
for url in "${CANDIDATES[@]}"; do
if curl -s --connect-timeout 1 "$url/api/tags" &>/dev/null; then
FOUND+=("$url")
fi
done
if [ ${#FOUND[@]} -eq 0 ]; then
# Ei löytynyt — yritä käynnistää lokaali
if command -v ollama &>/dev/null; then
echo " Käynnistetään Ollama..."
ollama serve &>/dev/null &
sleep 3
if curl -s --connect-timeout 1 "http://localhost:11434/api/tags" &>/dev/null; then
OLLAMA_URL="http://localhost:11434"
echo " ✓ Ollama käynnistetty ($OLLAMA_URL)"
else
echo " ✗ Ollaman käynnistys epäonnistui."
exit 1
fi
else
echo ""
echo " ✗ Ollamaa ei löytynyt."
echo " Kontti/remote: OLLAMA_URL=http://HOST:11434 ./kipina-node"
echo " Asenna: curl -fsSL https://ollama.ai/install.sh | sh"
exit 1
fi
elif [ ${#FOUND[@]} -eq 1 ]; then
OLLAMA_URL="${FOUND[0]}"
echo " ✓ Ollama löytyi: $OLLAMA_URL"
else
echo ""
echo " Löytyi ${#FOUND[@]} Ollama-instanssia:"
echo ""
for i in "${!FOUND[@]}"; do
echo " $((i+1))) ${FOUND[$i]}"
done
echo ""
read -p " Valitse [1-${#FOUND[@]}]: " -r CHOICE
if [[ "$CHOICE" =~ ^[0-9]+$ ]] && [ "$CHOICE" -ge 1 ] && [ "$CHOICE" -le ${#FOUND[@]} ]; then
OLLAMA_URL="${FOUND[$((CHOICE-1))]}"
else
OLLAMA_URL="${FOUND[0]}"
echo " Käytetään oletusta: $OLLAMA_URL"
fi
echo " ✓ Valittu: $OLLAMA_URL"
fi
echo ""
echo " Hub: $HUB_URL"
echo " Ollama: $OLLAMA_URL"
echo " Malli: $MODEL"
# Lataa malli (toimii sekä lokaalilla binäärillä että API:n kautta)
if ! curl -s "$OLLAMA_URL/api/tags" | grep -q "$MODEL"; then
echo " Ladataan $MODEL..."
curl -s "$OLLAMA_URL/api/pull" -d "{\"name\":\"$MODEL\"}" > /dev/null
fi
echo " ✓ Malli $MODEL valmis"
# Lataa binääri
BIN_PATH="./kipina-node-bin"
if [ ! -f "$BIN_PATH" ]; then
echo " Ladataan $BINARY..."
curl -sSL "$BASE_URL/$BINARY" -o "$BIN_PATH"
chmod +x "$BIN_PATH"
fi
echo ""
echo " ✓ Yhdistetään laskentaverkkoon..."
echo " Ctrl+C pysäyttää"
echo ""
HUB_URL="$HUB_URL" OLLAMA_URL="$OLLAMA_URL" OLLAMA_MODEL="$MODEL" exec "$BIN_PATH"

View File

@@ -10,6 +10,39 @@
<span id="compute-dot" class="status-dot" style="background:#30363d"></span>
<span style="color:#8b949e">Laskenta:</span>
<span id="compute-label" style="color:#8b949e">—</span>
<button id="compute-btn" class="btn btn-accent" title="Käynnistä kielimalli">Alusta</button>
<button id="compute-btn" class="btn btn-accent" title="Käynnistä kielimalli selaimessa">Alusta</button>
</span>
<span class="status-separator">│</span>
<span class="status-group">
<button id="join-btn" class="btn btn-green" onclick="showJoinDialog()" title="Liitä oma koneesi laskentaverkkoon (natiivi, nopea)">+ Liitä koneesi</button>
</span>
</div>
<!-- Join-dialogi -->
<div id="join-dialog" style="display:none;margin-top:8px;padding:16px;background:var(--panel);border:1px solid var(--border);border-radius:6px;font-size:14px">
<div style="display:flex;justify-content:space-between;align-items:center;margin-bottom:12px">
<span style="color:#e6edf3;font-weight:600;font-size:16px">Liitä koneesi laskentaverkkoon</span>
<button onclick="document.getElementById('join-dialog').style.display='none'" style="background:none;border:none;color:#8b949e;cursor:pointer;font-size:18px">✕</button>
</div>
<p style="color:#8b949e;margin-bottom:16px">Koneesi suorittaa tehtäviä ~10-50x nopeammin kuin selainlaskenta. Kaksi vaihetta:</p>
<!-- Vaihe 1: Ollama -->
<div style="margin-bottom:14px;padding:12px;background:var(--bg);border-radius:4px;border-left:3px solid var(--accent)">
<div style="color:#e6edf3;font-weight:600;margin-bottom:6px">1. Asenna Ollama <span style="color:#8b949e;font-weight:normal">(kielimallimoottori)</span></div>
<div style="display:flex;gap:6px;align-items:center;margin-bottom:6px">
<code style="flex:1;background:#010409;padding:8px 12px;border-radius:4px;color:var(--green);font-family:'Courier New',monospace;font-size:13px;user-select:all">curl -fsSL https://ollama.ai/install.sh | sh</code>
<button onclick="navigator.clipboard.writeText('curl -fsSL https://ollama.ai/install.sh | sh');this.textContent='✓';setTimeout(()=>this.textContent='Kopioi',1500)" class="btn btn-accent" style="padding:6px 10px">Kopioi</button>
</div>
<div style="color:#8b949e;font-size:12px">macOS: <code style="color:var(--accent)">brew install ollama</code> · Windows: <a href="https://ollama.ai/download" target="_blank" style="color:var(--accent)">ollama.ai/download</a> · Jos jo asennettu → siirry vaiheeseen 2.</div>
</div>
<!-- Vaihe 2: Kipinä-node -->
<div style="padding:12px;background:var(--bg);border-radius:4px;border-left:3px solid var(--green)">
<div style="color:#e6edf3;font-weight:600;margin-bottom:6px">2. Käynnistä Kipinä-node</div>
<div style="display:flex;gap:6px;align-items:center;margin-bottom:6px">
<code style="flex:1;background:#010409;padding:8px 12px;border-radius:4px;color:var(--green);font-family:'Courier New',monospace;font-size:13px;user-select:all">curl -sSL https://kipina.studio/kipina-node -o kipina-node && chmod +x kipina-node && ./kipina-node</code>
<button onclick="navigator.clipboard.writeText('curl -sSL https://kipina.studio/kipina-node -o kipina-node && chmod +x kipina-node && ./kipina-node');this.textContent='✓';setTimeout(()=>this.textContent='Kopioi',1500)" class="btn btn-green" style="padding:6px 10px">Kopioi</button>
</div>
<div style="color:#8b949e;font-size:12px">Lataa kielimallin (~2GB) automaattisesti ensimmäisellä kerralla. Ctrl+C pysäyttää.</div>
</div>
</div>

View File

@@ -53,6 +53,11 @@ import Settings from "../components/Settings.astro";
<script is:inline>
// === Helpers ===
window.showJoinDialog = function() {
const d = document.getElementById('join-dialog');
d.style.display = d.style.display === 'none' ? 'block' : 'none';
};
function esc(str) {
if (!str) return '';
return String(str).replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/"/g,'&quot;');
@@ -66,7 +71,7 @@ import Settings from "../components/Settings.astro";
// === Globaalit tilat ===
const defaultAgents = {
manager: { name: 'Manageri', avatar: '/avatars/karhunpentu.png', model: 'qwen-coder', order: 0,
manager: { name: 'Manageri', avatar: '/avatars/karhunpentu.webp', model: 'qwen-coder', order: 0,
temperature: 0.5, topK: 40, repeatPenalty: 1.15, maxTokens: 512,
prompt: `You are a senior project manager and software architect. Your job is to plan the file structure of a software project.
@@ -83,7 +88,7 @@ models.py: SQLAlchemy database models and engine setup
schemas.py: Pydantic request/response schemas
main.py: FastAPI application with CRUD endpoints
pyproject.toml: project dependencies` },
coder: { name: 'Koodari', avatar: '/avatars/kipina_notext.png', model: 'qwen-coder', order: 1,
coder: { name: 'Koodari', avatar: '/avatars/kipina_notext.webp', model: 'qwen-coder', order: 1,
temperature: 0.7, topK: 40, repeatPenalty: 1.15, maxTokens: 1024,
prompt: `You are an expert Python developer. Write complete, production-ready code.
@@ -104,7 +109,7 @@ NEVER:
- Forget to import from other project files
- Use requirements.txt or Poetry — always use pyproject.toml with [project] format (PEP 621)
- Use pip install — use uv (e.g. uv run uvicorn main:app --reload)` },
data: { name: 'Data', avatar: '/avatars/pesukarhu_notext.png', model: 'qwen-coder', order: 2,
data: { name: 'Data', avatar: '/avatars/pesukarhu_notext.webp', model: 'qwen-coder', order: 2,
temperature: 0.5, topK: 40, repeatPenalty: 1.15, maxTokens: 1024,
prompt: `You are a database architect specializing in SQLAlchemy and relational databases.
@@ -121,7 +126,7 @@ ALWAYS INCLUDE:
- from sqlalchemy.ext.declarative import declarative_base
- from sqlalchemy.orm import sessionmaker
- DATABASE_URL, engine, SessionLocal, Base` },
qa: { name: 'QA', avatar: '/avatars/susi_notext.png', model: 'qwen-coder', order: 3,
qa: { name: 'QA', avatar: '/avatars/susi_notext.webp', model: 'qwen-coder', order: 3,
temperature: 0.4, topK: 40, repeatPenalty: 1.15, maxTokens: 1024,
prompt: `You are a QA engineer writing automated tests.
@@ -138,7 +143,7 @@ TEST STRUCTURE:
5. test_delete: DELETE → 204, verify GET returns 404 after
ALWAYS: from fastapi.testclient import TestClient` },
tester: { name: 'DevOps', avatar: '/avatars/laiskiainen_notext.png', model: 'qwen-coder', order: 4,
tester: { name: 'DevOps', avatar: '/avatars/laiskiainen_notext.webp', model: 'qwen-coder', order: 4,
temperature: 0.3, topK: 40, repeatPenalty: 1.1, maxTokens: 512,
prompt: `You are a strict code reviewer and static analysis expert. Analyze the code line by line.
@@ -157,7 +162,7 @@ RESPOND:
- If all checks pass: "LGTM"
- If issues found: list each as "ISSUE: filename.py: description"
- Be specific and actionable, not vague` },
observer: { name: 'Tarkkailija', avatar: '/avatars/aikuinen_susi.png', model: 'qwen-coder', order: 5,
observer: { name: 'Tarkkailija', avatar: '/avatars/aikuinen_susi.webp', model: 'qwen-coder', order: 5,
temperature: 0.6, topK: 40, repeatPenalty: 1.15, maxTokens: 512,
prompt: `You are an independent technical observer and risk analyst.
@@ -315,7 +320,7 @@ OUTPUT FORMAT:
// Uuden agentin luonti
window.addCustomAgent = function() {
const id = 'custom_' + Date.now();
const avatars = ['/avatars/bear.png','/avatars/beaver.png','/avatars/gecko.png','/avatars/lion.png','/avatars/penguin.png','/avatars/spider.png','/avatars/walrus.png','/avatars/serpent.png'];
const avatars = ['/avatars/bear.webp','/avatars/beaver.webp','/avatars/gecko.webp','/avatars/lion.webp','/avatars/penguin.webp','/avatars/spider.webp','/avatars/walrus.webp','/avatars/serpent.webp'];
agents[id] = {
name: 'Uusi agentti',
avatar: avatars[Math.floor(Math.random() * avatars.length)],

View File

@@ -14,19 +14,20 @@
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
font-size: 16px;
background: var(--bg);
color: var(--text);
min-height: 100vh;
}
.container { max-width: 1200px; margin: 0 auto; padding: 20px; }
.container { max-width: 1600px; margin: 0 auto; padding: 20px 40px; }
/* Tabs */
.tabs { display: flex; gap: 4px; margin-bottom: 16px; }
.tab {
padding: 8px 16px; border-radius: 6px 6px 0 0; cursor: pointer;
padding: 10px 20px; border-radius: 6px 6px 0 0; cursor: pointer;
border: 1px solid var(--border); border-bottom: none;
background: var(--bg); color: #8b949e; font-size: 14px;
background: var(--bg); color: #8b949e; font-size: 15px;
}
.tab.active { background: var(--panel); color: var(--accent); border-color: var(--border); }
@@ -37,9 +38,9 @@ body {
/* Status bar */
.status-bar {
display: flex; align-items: center; gap: 12px;
padding: 8px 14px; background: var(--bg);
padding: 10px 16px; background: var(--bg);
border: 1px solid var(--border); border-radius: 6px 6px 0 0;
font-family: 'Courier New', monospace; font-size: 13px;
font-family: 'Courier New', monospace; font-size: 14px;
}
.status-dot {
width: 8px; height: 8px; border-radius: 50%; display: inline-block;
@@ -50,21 +51,22 @@ body {
/* Terminal */
.terminal {
background: #010409; border: 1px solid var(--border); border-top: none;
font-family: 'Courier New', monospace; font-size: 14px;
min-height: 300px; max-height: 60vh; overflow-y: auto;
padding: 8px 12px;
font-family: 'Courier New', monospace; font-size: 16px;
min-height: 400px; max-height: 70vh; overflow-y: auto;
padding: 12px 16px;
}
.terminal-line { padding: 1px 0; white-space: pre-wrap; word-break: break-word; }
.terminal-prompt { color: var(--yellow); margin-right: 8px; }
.terminal-input-row {
display: flex; align-items: center; position: relative;
background: #010409; border: 1px solid var(--border); border-top: none;
border-radius: 0 0 6px 6px; padding: 8px 12px;
font-family: 'Courier New', monospace; font-size: 14px;
background: #0d1117; border: 1px solid var(--accent); border-top: none;
border-radius: 0 0 6px 6px; padding: 10px 14px;
font-family: 'Courier New', monospace; font-size: 15px;
box-shadow: 0 2px 8px rgba(88,166,255,0.1);
}
.terminal-input {
flex: 1; background: transparent; border: none; outline: none;
color: var(--green); font-family: inherit; font-size: inherit;
color: var(--green); font-family: inherit; font-size: 16px;
}
.terminal-dropdown {
display: none; position: absolute; bottom: 100%; left: 30px;
@@ -128,10 +130,10 @@ body {
background: linear-gradient(145deg, rgba(33,38,45,0.4) 0%, rgba(13,17,23,0.8) 100%);
backdrop-filter: blur(12px);
border: 1px solid rgba(240,246,252,0.1);
border-radius: 12px;
padding: 6px 6px 4px;
border-radius: 14px;
padding: 8px 8px 6px;
text-align: center;
width: 72px;
width: 90px;
opacity: 0.8;
cursor: pointer;
transition: all 0.4s cubic-bezier(0.175, 0.885, 0.32, 1.275);
@@ -144,12 +146,12 @@ body {
box-shadow: 0 8px 14px rgba(0,0,0,0.4);
}
.agent-avatar img {
width: 50px; height: 50px; border-radius: 12px;
width: 64px; height: 64px; border-radius: 14px;
margin-bottom: 4px; border: 2px solid rgba(240,246,252,0.1);
transition: all 0.4s ease; object-fit: cover;
}
.agent-avatar .avatar-name {
font-size: 10px; color: #8b949e; white-space: nowrap;
font-size: 11px; color: #8b949e; white-space: nowrap;
overflow: hidden; text-overflow: ellipsis;
}
.agent-avatar.active {

View File

@@ -1,6 +1,6 @@
[package]
name = "hub"
version = "0.3.0"
version = "0.3.1"
edition = "2024"
[dependencies]

View File

@@ -34,7 +34,7 @@ struct AppState {
total_tasks: Mutex<u64>,
stats_tx: broadcast::Sender<String>,
node_channels: tokio::sync::RwLock<HashMap<u64, tokio::sync::mpsc::UnboundedSender<String>>>, // Kohdennettu reititys
pending_consensus: tokio::sync::RwLock<HashMap<String, Vec<serde_json::Value>>>, // Proof of Compute -konsensus
_pending_consensus: tokio::sync::RwLock<HashMap<String, Vec<serde_json::Value>>>, // Proof of Compute -konsensus
feature_flags: tokio::sync::RwLock<HashMap<String, bool>>, // Tuntee TODO.md:n ruksit lennosta
ip_connections: Mutex<HashMap<IpAddr, u32>>,
node_ips: Mutex<HashMap<u64, IpAddr>>,
@@ -256,7 +256,7 @@ async fn main() {
total_tasks: Mutex::new(0),
stats_tx: stats_tx.clone(),
node_channels: tokio::sync::RwLock::new(HashMap::new()),
pending_consensus: tokio::sync::RwLock::new(HashMap::new()),
_pending_consensus: tokio::sync::RwLock::new(HashMap::new()),
feature_flags: tokio::sync::RwLock::new(HashMap::new()),
ip_connections: Mutex::new(HashMap::new()),
node_ips: Mutex::new(HashMap::new()),
@@ -502,7 +502,12 @@ async fn ws_handler(
) -> impl IntoResponse {
// Origin-tarkistus — estää cross-site WebSocket hijackingin
if let Some(origin) = headers.get("origin").and_then(|v| v.to_str().ok()) {
if !ALLOWED_ORIGINS.iter().any(|&allowed| origin == allowed) {
let is_allowed = ALLOWED_ORIGINS.iter().any(|&allowed| origin == allowed)
|| origin.starts_with("http://192.168.")
|| origin.starts_with("http://10.")
|| origin.starts_with("http://172."); // LAN-avaruudet
if !is_allowed {
tracing::warn!("Estetty yhteys väärällä originilla: {}", origin);
return (
axum::http::StatusCode::FORBIDDEN,
@@ -1099,7 +1104,7 @@ async fn api_chat_completions(
// Etsitään vapaa solmu — priorisoidaan natiivisolmut (GPU) selaimen edelle
let (target_node, _total_matching) = {
let tasks = state.node_tasks.lock().unwrap();
let busy = state.node_busy.lock().unwrap();
let _busy = state.node_busy.lock().unwrap();
let node_types = state.node_types.lock().unwrap();
let matching: Vec<u64> = tasks.iter().filter(|(_, task)| {
// Eksakti match tai qwen-perheen yhteensopivuus (selain: qwen-coder-05b, natiivi: qwen2.5-coder:7b)

View File

@@ -347,22 +347,26 @@ async fn main() {
if let Some(ref engine) = llm {
let max_tokens = task.get("max_tokens").and_then(|v| v.as_u64()).unwrap_or(1024) as usize;
tracing::info!("Generoidaan (task_id: {}, max_tokens: {}): \"{}\"", task_id, max_tokens, &prompt[..prompt.len().min(100)]);
let prompt_lines = prompt.lines().count();
let prompt_last: String = prompt.lines().last().unwrap_or("").chars().take(60).collect();
tracing::info!("→ task_id:{} | {}r prompti | \"{}...\"", task_id, prompt_lines, prompt_last);
let model_name = engine.model_name();
match engine.generate(prompt, max_tokens).await {
Ok(result) => {
tracing::info!(
"Tulos: {} tokenia | {:.0}ms | {:.1} tok/s | \"{}\"",
"✓ {} | {} tok | {:.0}ms | {:.1} tok/s",
model_name,
result.tokens_generated,
result.duration_ms,
result.tokens_per_sec,
&result.text[..result.text.len().min(80)]
);
// Lähetetään vain lyhyt prompti-esikatselu (ei koko kontekstia)
let prompt_short: String = prompt.lines().last().unwrap_or("").chars().take(100).collect();
let done = json!({
"type": "llm_done",
"prompt": prompt,
"prompt": prompt_short,
"model": format!("{} (Ollama)", model_name),
"response": result.text,
"tokens_generated": result.tokens_generated,

View File

@@ -348,7 +348,9 @@ pub async fn start_agent_node(hub_url: String, has_webgpu: bool, device_info_jso
});
}
}
} else if msg.contains("llm_prompt") && (current_task == 4 || current_task == 5) {
} else if msg.contains("llm_prompt") {
console_log!("[DEBUG] llm_prompt vastaanotettu! current_task={}, busy={}", current_task, LLM_BUSY.load(Ordering::SeqCst));
if current_task == 4 || current_task == 5 {
// Qwen2.5-Coder: 4 = 0.5B, 5 = 3B
if let Ok(task) = serde_json::from_str::<serde_json::Value>(&msg) {
let prompt = task.get("prompt").and_then(|v| v.as_str()).unwrap_or("").to_string();
@@ -376,6 +378,7 @@ pub async fn start_agent_node(hub_url: String, has_webgpu: bool, device_info_jso
}
}
}
} // current_task == 4 || 5
} else if msg.contains("ai_task") {
console_log!("Hub task vastaanotettu, ajetaan GPU:lla...");
let ws_for_async = ws_clone.clone();

View File

@@ -248,14 +248,17 @@ async fn get_or_build_model(use_3b: bool, ws: &Rc<RefCell<WebSocket>>) -> Result
/// use_3b: false = 0.5B (nopea), true = 3B (laadukas)
pub async fn run_coder_inference(prompt: String, ws: Rc<RefCell<WebSocket>>, use_3b: bool, task_id: Option<String>) {
console_log!("[Coder] run_coder_inference alkaa! prompt={}", &prompt[..prompt.len().min(50)]);
let size_label = if use_3b { "3B" } else { "0.5B" };
let start_load = crate::perf_now();
console_log!("[Coder] Kutsutaan get_or_build_model...");
if let Err(e) = get_or_build_model(use_3b, &ws).await {
console_log!("[Coder] Mallin lataus: {}", e);
console_log!("[Coder] Mallin lataus epäonnistui: {}", e);
return;
}
console_log!("[Coder] Malli valmis, aloitetaan inferenssi");
let load_time = crate::perf_now() - start_load;
if load_time > 100.0 {

View File

@@ -1,413 +0,0 @@
# Kipinä Agentic Studio — Opas
Hajautettu AI-laskentaverkko jossa kielimallit ajavat koodia suoraan selaimessa.
Tämä opas selittää miten kielimallit toimivat, miten niitä ohjataan, ja miten
tuloksia voi parantaa.
---
## Kielimallit ja niiden koot
Kielimalli on neuroverkko joka ennustaa seuraavan sanan (tokenin) edellisten
perusteella. Mallin "koko" tarkoittaa parametrien (painojen) määrää:
| Malli | Parametrit | Koko levyllä | Nopeus selaimessa | Koodinlaatu |
|-------|-----------|-------------|-------------------|-------------|
| SmolLM 135M | 135 miljoonaa | ~270 MB | ~5 tok/s | Yksinkertainen teksti |
| Qwen2.5-Coder:0.5B | 500 miljoonaa | ~990 MB | ~3-6 tok/s | Pienet funktiot |
| Qwen2.5-Coder:3B | 3 miljardia | ~6.2 GB | ~0.4 tok/s | Kokonaiset tiedostot |
| GPT-4 (vertailu) | ~1800 miljardia | ~3.6 TB | pilvipalvelu | Kokonaiset projektit |
**Parametrien vaikutus:** Jokainen parametri on yksi liukuluku (float16 = 2 tavua)
joka tallentaa opittua tietoa. 0.5B-malli tietää perusrakenteet mutta tekee
loogisia virheitä. 3B-malli ymmärtää kontekstin paremmin. Ero on kuin sanakirjan
ja oppikirjan välillä.
**Miksi selaimessa?** Malli ajetaan käyttäjän omalla laitteella WebAssemblyn
kautta. Data ei lähde koneelta, eikä tarvita pilvipalvelua. Haittapuoli on
hitaus — GPU-palvelimella sama 0.5B-malli tuottaa ~100 tok/s.
---
## Tokenit — kielimallin "sanat"
Malli ei näe tekstiä kirjaimina vaan **tokeneina**. Tokeni on yleensä
sanan osa, kokonainen sana tai välilyönti. Tokenisaatio tehdään
BPE-algoritmilla (Byte Pair Encoding) joka oppii yleisimmät
merkkijonot harjoitusdatasta.
### Esimerkki: suomi vs. englanti
Alla oikea tokenisointitulos Qwen2.5-Coder-tokenisaattorilla. Jokainen
värikoodattu lohko on yksi tokeni — huomaa miten suomi vaatii enemmän
tokeneita saman merkityksen välittämiseen:
![Tokenisointivertailu EN/FI](/images/tokenization-example.png)
**Huomaa miten:**
- Englannin yleiset sanat (`the`, `in`, `a`, `function`) ovat kokonaisia tokeneita
- Suomen sanat pilkotaan pienempiin osiin (`Hajautettu` → 4 tokenia, `Distributed` → 2)
- Suomi vaatii **30-50% enemmän tokeneita** saman merkityksen välittämiseen
- Koodiavainsanat (`function`, `list`, `sort`) ovat tehokkaita molemmilla kielillä
### Miksi tämä merkitsee?
**Jokainen tokeni = yksi laskentakierros.** Jos suomi vaatii 50% enemmän tokeneita:
1. **Hitaampi vastaus:** 100 tokenin englanninkielinen vastaus ≈ 150 tokenia suomeksi
→ 50% pidempi odotusaika
2. **Pienempi konteksti:** Sama merkityssisältö vie enemmän tilaa konteksti-ikkunasta
3. **Huonompi ymmärrys:** Pitkät sanat pilkotaan osiin jotka malli ei välttämättä
tunnista → hallusinaatiot lisääntyvät
**Siksi tekniset promptit ovat englanniksi** — malli saa enemmän informaatiota
samassa token-budjetissa ja ymmärtää ohjeet paremmin.
**Token-budjetti tässä järjestelmässä:**
| Osa | Tokeneita | Osuus |
|-----|-----------|-------|
| System prompt | ~30 | kiinteä |
| Agent prompt | ~25 | kiinteä |
| Konteksti (aiemmat tiedostot) | 0-300 | kasvaa |
| Käyttäjän prompti | ~20-50 | vaihtelee |
| **Syöte yhteensä** | **~75-400** | |
| Generoitu vastaus (max) | 512 | raja |
| **Yhteensä** | **~600-900** | /32 768 |
Konteksti-ikkuna on reilusti riittävä. Pullonkaula ei ole ikkunan koko
vaan **mallin kyky ymmärtää pitkää kontekstia** — 0.5B-malli alkaa
"unohtaa" ohjeet kun konteksti kasvaa yli ~200 tokenin.
---
## Promptit — miten mallia ohjataan
### Kolmitasoinen prompttirakenne
```mermaid
flowchart TD
S["System prompt<br/><i>You are a coding assistant. Respond with ONLY code.</i><br/>🔒 Kiinteä, kovakoodattu — malli priorisoi tämän"]
A["Agent prompt<br/><i>Olet kokenut ohjelmistokehittäjä...</i><br/>✏️ Käyttäjän muokattavissa UI:ssa"]
U["User prompt<br/><i>Write ONLY the file main.py...</i><br/>📋 Vaihtelee joka kutsussa, sisältää kontekstin"]
P["Prefill: ``` <br/>🎯 Pakottaa mallin aloittamaan koodilla"]
S --> A --> U --> P
P -->|malli jatkaa| R["Generoitu koodi"]
style S fill:#1a1e2e,stroke:#f85149,color:#c9d1d9
style A fill:#1a1e2e,stroke:#d29922,color:#c9d1d9
style U fill:#1a1e2e,stroke:#3fb950,color:#c9d1d9
style P fill:#1a1e2e,stroke:#a371f7,color:#c9d1d9
style R fill:#0d1117,stroke:#58a6ff,color:#58a6ff
```
### Miksi promptit ovat englanniksi?
Qwen2.5-Coder on harjoitettu pääosin englanninkielisellä koodilla ja
dokumentaatiolla. Suomenkielinen ohje kuluttaa enemmän tokeneita JA
malli ymmärtää sen huonommin. Agenttien nimet ja käyttöliittymä ovat
suomeksi, mutta tekniset ohjeet mallille englanniksi.
Poikkeus: agenttipromptit ovat suomeksi koska ne menevät user-blokkiin
(ei system-blokkiin) ja niiden tarkoitus on enemmän "persoonallisuus"
kuin tekninen ohje.
---
## Prefill-tekniikka
Normaalisti malli päättää vapaasti miten vastaa:
```
Ilman prefilliä:
Malli: "Sure! Here is a Python program that prints Hello World:\n```python\nprint('Hello')\n```"
→ 25 tokenia, joista 15 turhia
Prefillin kanssa:
Me syötämme: ```
Malli jatkaa: python\nprint('Hello')\n```
→ 5 tokenia, kaikki hyödyllisiä
```
Prefill on kuin aloittaisit lauseen toisen puolesta — malli jatkaa
siitä mihin jäit sen sijaan, että aloittaisi kohteliaalla johdannolla.
**Sivuvaikutus:** Malli tuottaa kielitunnisteen (`python`, `rust`) ja
sulkevan ` ``` `:n. Nämä siivotaan jälkikäteen `strip_markdown_wrapper`-funktiolla.
---
## Sampling — miten malli valitsee seuraavan tokenin
Malli ei "tiedä" oikeaa vastausta. Se laskee jokaiselle mahdolliselle
seuraavalle tokenille todennäköisyyden ja valitsee yhden. Valintaa
ohjataan kolmella parametrilla:
### Temperature (0.7)
Kontrolloi "luovuutta" vs. "varmuutta":
```
Temperature 0.0 (greedy): Aina todennäköisin tokeni → "def fibonacci(n):"
Temperature 0.7 (oletus): Painottaa todennäköisiä mutta sallii vaihtelua
Temperature 1.5 (luova): Lähes satunnainen → "async lambda fib = ..."
```
0.7 on kompromissi: tarpeeksi determinististä tuottamaan toimivaa koodia,
mutta tarpeeksi vaihtelevaa välttämään toistoa.
### Top-k (40)
Rajaa valinnan 40 todennäköisimpään tokeniin. Estää mallia valitsemasta
täysin absurdeja vaihtoehtoja:
```
Ilman top-k: 150 936 vaihtoehtoa → voi valita minkä tahansa
Top-k 40: 40 vaihtoehtoa → järkevät vaihtoehdot
Top-k 1: 1 vaihtoehto → greedy (aina sama vastaus)
```
### Repetition penalty (1.15)
Vähentää jo tuotettujen tokenien todennäköisyyttä. Estää mallia
juuttumasta luuppiin:
```
Ilman rangaistusta: "print print print print print..."
Penalty 1.15: "print('Hello')\nprint('World')"
```
1.15 on lievä rangaistus — estää pahimman toiston mutta sallii
saman avainsanan (esim. `return`) esiintymisen useasti.
---
## Stop-sekvenssit — milloin generointi loppuu
Malli generoi tokeneita kunnes jokin näistä tapahtuu:
1. **EOS-tokeni** (151645): Mallin oma "loppu"-merkki
2. **Max tokens** (512): Kovakoodattu raja
3. **Stop-sekvenssi**: Malli alkaa tuottaa selitystä
```
fn fibonacci(n: usize) -> usize {
if n <= 1 { return n; }
fibonacci(n-1) + fibonacci(n-2)
}
← Tähän asti koodia, ok
// Example usage: ← Stop! Tämä ei ole enää vastausta
let result = fibonacci(10); ← Ei generoida
```
Tunnistetut stop-sekvenssit: `### `, `Explanation`, `Note:`, `Output:`,
`// Example`, `# Example`. Generointi katkaistaan ja teksti trimmataan
stop-kohtaan.
---
## Projekti-pipeline — miten agenttitiimi toimii
```mermaid
flowchart TD
U["Käyttäjä: FastAPI + SQLite REST API for users"] --> M
M["🟡 Manageri: Pilko tiedostoiksi"] -->|tiedostolista| C1
C1["🟢 Koodari: models.py"] -->|"konteksti: models.py"| C2
C2["🟢 Koodari: main.py"] -->|"konteksti: models + main"| C3
C3["🟢 Koodari: pyproject.toml"] -->|kaikki tiedostot| T1
T1["🔵 Testaaja: Review"] -->|bugeja löytyi| C4
T1 -->|LGTM| Done["✅ Projekti valmis"]
C4["🟡 Koodari: Korjaukset"] --> T2
T2["🔵 Testaaja: Uudelleenarviointi"] --> Done
```
**Kontekstin ketjutus** on kriittistä: kun koodari kirjoittaa `main.py`:tä,
se saa `models.py`:n sisällön promptissa. Ilman tätä se ei tietäisi
mitä luokkia importata.
**Riippuvuusjärjestys:** Manageria pyydetään listaamaan riippuvuudet ensin
(models.py ennen main.py) jotta kontekstiketju toimii oikeaan suuntaan.
---
## Laadun parantaminen
### 1. Isompi malli (suurin vaikutus)
| | 0.5B | 3B | Pilvi-API |
|---|---|---|---|
| Fibonacci | Joskus virheitä | Yleensä oikein | Aina oikein |
| FastAPI CRUD | Voi käyttää Flaskia | Oikea kirjasto | Täydellinen |
| Monimutkainen logiikka | Hallusinoi | Osaa perusasiat | Syvä ymmärrys |
| Nopeus (selain) | ~5 tok/s | ~0.4 tok/s | — |
| Latauksen koko | 990 MB | 6.2 GB | 0 (API) |
**Käytännössä:** `kpn load 2` lataa 3B-mallin. Hitaampi mutta huomattavasti
parempi koodinlaatu. Suositus monimutkaisiin projekteihin.
### 2. Paremmat promptit (ilmaista)
**Huono:** `"tee fibonacci"`
- Malli ei tiedä kieltä, formaattia tai kontekstia
**Hyvä:** `"Write a fibonacci function in Rust that returns Vec<u64>"`
- Kieli, palautustyyppi ja rakenne määritelty
**Promptin säännöt:**
- Englanniksi (tehokkaampi tokenisointi, parempi ymmärrys)
- Konkreettinen (mainitse kieli, kirjastot, palautustyyppi)
- Lyhyt (jokainen sana kuluttaa tokenin konteksti-ikkunasta)
- Positiivinen ("Write X" ei "Don't write Y")
### 3. Kontekstin hallinta (pipeline-taso)
**Ongelma:** 0.5B-malli "unohtaa" promptin alun kun konteksti kasvaa.
**Ratkaisu:** Pienet, kohdennetut promptit:
- Yksi tiedosto kerrallaan (ei "kirjoita koko projekti")
- Vain relevantit aiemmat tiedostot kontekstina
- Max 4 tiedostoa per projekti
### 4. Iterointi (review-luuppi)
Yksi generointikierros tuottaa harvoin virheetöntä koodia.
Pipeline-arkkitehtuuri mahdollistaa:
1. **Generointi** — ensimmäinen versio
2. **Review** — testaaja löytää ongelmat
3. **Korjaus** — koodari saa palautteen ja korjaa
4. **Uusi review** — tarkistetaan korjaukset
Nykyinen järjestelmä tekee max 1 korjauskierroksen. Useampi
iteraatio parantaisi laatua mutta kasvattaisi laskenta-aikaa.
### 5. Erikoistetut system promptit
Oletuspromptit ovat yleiskäyttöisiä. Projektikohtaiset promptit
parantavat laatua merkittävästi:
```
Oletus: "Olet kokenut ohjelmistokehittäjä."
Parempi: "You are a Python backend developer specializing in FastAPI.
Always use Pydantic models for request/response schemas.
Always use dependency injection for database sessions.
Follow the repository pattern."
```
Agenttikohtaiset promptit voi muokata suoraan UI:ssa.
### 6. Few-shot esimerkit
Malli oppii parhaiten esimerkeistä. Sen sijaan, että sanot "kirjoita
FastAPI endpoint", näytä miltä haluat tuloksen näyttävän:
```
Write a GET endpoint like this example:
@app.get("/items")
def list_items():
db = SessionLocal()
return db.query(Item).all()
Now write a similar endpoint for /users.
```
0.5B-malli jäljittelee rakennetta tehokkaasti — se on parempi kopioimaan
kuin keksimään. Nykyinen pyproject.toml-esimerkki promptissa on tätä tekniikkaa.
### 7. Temperature-säätö tehtävän mukaan
Nykyinen temperature 0.7 on kompromissi. Eri tehtävät hyötyisivät eri arvoista:
| Tehtävä | Paras temperature | Miksi |
|---------|-------------------|-------|
| Tarkka koodi (CRUD, boilerplate) | 0.2-0.4 | Determinismi tärkeää |
| Luova koodi (algoritmit, arkkitehtuuri) | 0.6-0.8 | Vaihtelu löytää ratkaisuja |
| Vapaa teksti (kommentit, dokumentaatio) | 0.8-1.0 | Luonnollisempi kieli |
Järjestelmä voisi valita temperaturen automaattisesti tehtävätyypin perusteella.
### 8. Ensemble — sama prompti usealle mallille
Lähetetään sama tehtävä kahdelle solmulle ja valitaan parempi vastaus.
Nykyinen Proof of Compute -arkkitehtuuri tukee tätä periaatteessa:
hub voisi reitittää saman task_id:n kahdelle solmulle ja verrata tuloksia.
Käytännössä tämä kaksinkertaistaa laskenta-ajan mutta parantaa laatua
merkittävästi — virheellinen vastaus harvoin on sama kahdella ajolla
koska sampling on stokastinen.
### 9. Post-processing (nykyinen)
Mallin raakavastaus siivotaan:
1. Kielitunniste poistetaan (`python`, `rust`, ...)
2. Sulkeva ` ``` ` poistetaan
3. Johdantolauseet poistetaan ("Sure!", "Here is...")
4. Selityskommentit poistetaan ("# This is a simple...")
5. Stop-sekvenssit katkaisevat generoinnin
Tämä ei paranna mallin ajattelua mutta poistaa turhan roskan.
### 10. Mallin hienosäätö (fine-tuning)
Qwen2.5-Coder on yleiskäyttöinen koodimalli. Jos sitä hienosäätäisi
omalla koodiaineistolla (esim. yrityksen koodikanta, tietty framework),
se tuottaisi huomattavasti parempaa koodia juuri siihen kontekstiin.
LoRA-hienosäätö 0.5B-mallille vaatii ~4 GB GPU-muistia ja muutaman
tunnin harjoittelua. Tulos on erikoistunut malli joka osaa tuottaa
esimerkiksi juuri FastAPI + SQLAlchemy -koodia luotettavasti.
---
## Välimuistiarkkitehtuuri — miksi toinen lataus on nopea
```
Ensimmäinen lataus (hidas):
Verkko (HuggingFace CDN) → IndexedDB → RAM → Mallin rakennus
~990 MB lataus, ~30-60s
Toinen lataus samalla sivulatauksella (nopea):
RAM-cache → Mallia ei rakenneta uusiksi, vain KV-cache nollataan
~0ms
Refresh jälkeen (keskitaso):
IndexedDB → RAM → Mallin rakennus
~0 MB lataus, ~2-5s rakennus
Uusi selain/laite (hidas):
Verkko → IndexedDB → RAM → Mallin rakennus
Kuten ensimmäinen lataus
```
**KV-cache:** Mallin sisäinen muisti joka tallentaa aiempien tokenien
laskenta tulokset. Nollataan (`clear_kv_cache()`) jokaisen promptin
välillä jotta edellinen vastaus ei vuoda seuraavaan.
---
## Lukuja käytännöstä
**Yksittäinen funktio** (esim. fibonacci):
- Input: ~80 tokenia
- Output: ~50-100 tokenia
- Aika: ~10-20s (0.5B, selain)
- Laatu: Yleensä toimiva, joskus loogisia virheitä
**3 tiedoston projekti** (esim. FastAPI CRUD):
- Manageri: ~30 tok out
- Koodari (3x): ~100-150 tok out per tiedosto
- Testeri: ~50 tok out
- Korjaukset: ~100 tok out (jos tarpeen)
- **Yhteensä: ~500-700 tokenia, ~3-5 min**
- Laatu: Rakenne oikein, yksittäisiä bugeja
**Token-kustannus vs. pilvipalvelu:**
- Tässä järjestelmässä: 0 euroa (laskenta omalla koneella)
- GPT-4 API: ~700 tokenia x $0.03/1K = ~$0.02 per projekti
- Claude API: ~700 tokenia x $0.015/1K = ~$0.01 per projekti
Selaimessa ajettava malli on ilmainen mutta huomattavasti hitaampi
ja heikompilaatuinen kuin pilvi-API. Sopii oppimiseen, prototypointiin
ja tilanteisiin joissa data ei saa lähteä omalta koneelta.

View File

@@ -1,34 +0,0 @@
# Kipinä Agentic Playground - Animaatioiden käyttöönotto
Koska Kipinä-verkon agenttien avatarit tällä erää ovat staattisia PNG-kuvatiedostoja, käyttöliittymä hyödyntää CSS-pohjaista pomppimisilmiötä (sekä pulppuavaa 💬 puhekuplaa) "puhumisen" merkkinä. Olemme kuitenkin koodanneet taustalle piilotetun tuen aivioiduille videoloopeille myöhempää käyttöä varten!
Näin saat UI:n tukemaan oikeasti animoituja kasvoja/videoita.
## 1. Luo Animoidut GIF-tiedostot
Valitse mikä tahansa ulkoinen AI-työkalu (kuten HeyGen, Pika v1.0, tai Midjourney+Runway yhdistelmä) ja muunna avatar-kuvat (esim. `kettu_notext.png`) 3-5 sekunnin kestäviksi GIF-loopeiksi. Hahmon leuka tulisi pyöriä tai naama vääntyillä puhuessaan.
## 2. Nimeä Tiedostot Oikein ja Lisää Ne Kansioon
Siirrä uudet GIF-animaatiot samaan kansioon alkuperäisten kuvien kanssa. Muuta niiden nimi siten, että se päättyy tunnisteeseen `_puhuva.gif`.
Esimerkkejä:
- Koodari `kipina_notext.png``kipina_notext_puhuva.gif`
- Manageri `karhunpentu.png``karhunpentu_puhuva.gif`
- Asiakas `kettu_notext.png``kettu_notext_puhuva.gif`
## 3. Aktivoi Koodi
Käännä Kipinä Playground -ohjaimen JavaScript-koodista piilotettu ominaisuus päälle.
Etsi tiedostosta `../index.html` (noin riviltä 1084, `updatePromptEditor`-funktiosta):
```javascript
// Piilotettu ominaisuus: Puhuvien videoiden / gif-animaatioiden kytkentä
window.USE_ANIMATED_GIFS = false;
```
Muuta tuo `false` arvoon `true`:
```javascript
window.USE_ANIMATED_GIFS = true;
```
**Mitä logiikka tekee?**
Aina kun valitset agentin kaaviosta, koodi korvaa aktiivisen kuvakkeen lopussa olevan `.png` -päätteen sanalla `_puhuva.gif` lennosta! Jos poistut agentin valinnasta tai valitset jonkun toisen, koodi vaihtaa kuvan välittömästi takaisin staattiseen `.png`-versioon ja sulkee ilmentymän suun.
Näin saat kaikkien asiantuntijoiden face-track looppeja hallittua yhdellä kädenkäänteellä.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 696 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 757 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 700 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 731 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 711 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 695 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 130 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 432 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 650 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 389 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 596 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 496 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 872 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 738 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 109 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 130 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.4 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 813 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 658 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 593 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 563 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 696 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 718 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 513 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 780 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 826 KiB

View File

@@ -1,43 +0,0 @@
# OpenTofu Core Codebase Documentation
This directory contains some documentation about the OpenTofu Core codebase,
aimed at readers who are interested in making code contributions.
If you're looking for information on _using_ OpenTofu, please instead refer
to [the main OpenTofu CLI documentation](https://opentofu.org/docs/cli/index.html).
## OpenTofu Core Architecture Documents
* [OpenTofu Core Architecture Summary](./architecture.md): an overview of the
main components of OpenTofu Core and how they interact. This is the best
starting point if you are diving in to this codebase for the first time.
* [Resource Instance Change Lifecycle](./resource-instance-change-lifecycle.md):
a description of the steps in validating, planning, and applying a change
to a resource instance, from the perspective of the provider plugin RPC
operations. This may be useful for understanding the various expectations
OpenTofu enforces about provider behavior, either if you intend to make
changes to those behaviors or if you are implementing a new OpenTofu plugin
SDK and so wish to conform to them.
(If you are planning to write a new provider using the _official_ SDK then
please refer to [the Extend documentation](https://github.com/hashicorp/terraform-docs-common)
instead; it presents similar information from the perspective of the SDK
API, rather than the plugin wire protocol.)
* [Diagnostics](./diagnostics): how we report errors and warnings to end-users
in OpenTofu.
* [Plugin Protocol](./plugin-protocol/): gRPC/protobuf definitions for the
plugin wire protocol and information about its versioning strategy.
This documentation is for SDK developers, and is not necessary reading for
those implementing a provider using the official SDK.
* [How OpenTofu Uses Unicode](./unicode.md): an overview of the various
features of OpenTofu that rely on Unicode and how to change those features
to adopt new versions of Unicode.
## Contribution Guides
* [Contributing to OpenTofu](../CONTRIBUTING.md): a complete guideline for those who want to contribute to this project.

View File

@@ -1,374 +0,0 @@
# OpenTofu Core Architecture Summary
This document is a summary of the main components of OpenTofu Core and how
data and requests flow between these components. It's intended as a primer
to help navigate the codebase to dig into more details.
We assume some familiarity with user-facing OpenTofu concepts like
configuration, state, CLI workflow, etc. The OpenTofu website has
documentation on these ideas.
## OpenTofu Request Flow
The following diagram shows an approximation of how a user command is
executed in OpenTofu:
![OpenTofu Architecture Diagram, described in text below](./images/architecture-overview.png)
Each of the different subsystems (solid boxes) in this diagram is described
in more detail in a corresponding section below.
## CLI (`command` package)
Each time a user runs the `tofu` program, aside from some initial
bootstrapping in the root package (not shown in the diagram) execution
transfers immediately into one of the "command" implementations in
[the `command` package](https://pkg.go.dev/github.com/opentofu/opentofu/internal/command).
The mapping between the user-facing command names and
their corresponding `command` package types can be found in the `commands.go`
file under the `cmd/tofu` directory (package `main`).
The full flow illustrated above does not actually apply to _all_ commands,
but it applies to the main OpenTofu workflow commands `tofu plan` and
`tofu apply`, along with a few others.
For these commands, the role of the command implementation is to read and parse
any command line arguments, command line options, and environment variables
that are needed for the given command and use them to produce a
[`backend.Operation`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/backend#Operation)
object that describes an action to be taken.
An _operation_ consists of:
* The action to be taken (e.g. "plan", "apply").
* The name of the [workspace](https://opentofu.org/docs/language/state/workspaces)
where the action will be taken.
* Root module input variables to use for the action.
* For the "plan" operation, a path to the directory containing the configuration's root module.
* For the "apply" operation, the plan to apply.
* Various other less-common options/settings such as `-target` addresses, the
"force" flag, etc.
The operation is then passed to the currently-selected
[backend](https://opentofu.org/docs/language/settings/backends/configuration). Each backend name
corresponds to an implementation of
[`backend.Backend`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/backend#Backend), using a
mapping table in
[the `backend/init` package](https://pkg.go.dev/github.com/opentofu/opentofu/internal/backend/init).
Backends that are able to execute operations additionally implement
[`backend.Enhanced`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/backend#Enhanced);
the command-handling code calls `Operation` with the operation it has
constructed, and then the backend is responsible for executing that action.
Backends that execute operations, however, do so as an architectural implementation detail and not a
general feature of backends. That is, the term 'backend' as a OpenTofu feature is used to refer to
a plugin that determines where OpenTofu stores its state snapshots - only the default `local`, `remote` and `cloud` backends perform operations.
Thus, most backends do _not_ implement this interface, and so the `command` package wraps these
backends in an instance of
[`local.Local`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/backend/local#Local),
causing the operation to be executed locally within the `tofu` process itself.
## Backends
A _backend_ determines where OpenTofu should store its state snapshots.
As described above, the `local` backend also executes operations on behalf of most other
backends. It uses a _state manager_
(either
[`statemgr.Filesystem`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/states/statemgr#Filesystem) if the
local backend is being used directly, or an implementation provided by whatever
backend is being wrapped) to retrieve the current state for the workspace
specified in the operation, then uses the _config loader_ to load and do
initial processing/validation of the configuration specified in the
operation. It then uses these, along with the other settings given in the
operation, to construct a
[`tofu.Context`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#Context),
which is the main object that actually performs OpenTofu operations.
The `local` backend finally calls an appropriate method on that context to
begin execution of the relevant command, such as
[`Plan`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#Context.Plan)
or
[`Apply`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#Context.Apply), which in turn constructs a graph using a _graph builder_,
described in a later section.
## Configuration Loader
The top-level configuration structure is represented by model types in
[package `configs`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/configs).
A whole configuration (the root module plus all of its descendent modules)
is represented by
[`configs.Config`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/configs#Config).
The `configs` package contains some low-level functionality for constructing
configuration objects, but the main entry point is in the sub-package
[`configload`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/configs/configload]),
via
[`configload.Loader`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/configs/configload#Loader).
A loader deals with all of the details of installing child modules
(during `tofu init`) and then locating those modules again when a
configuration is loaded by a backend. It takes the path to a root module
and recursively loads all of the child modules to produce a single
[`configs.Config`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/configs#Config)
representing the entire configuration.
OpenTofu expects configuration files written in the OpenTofu language, which
is a DSL built on top of
[HCL](https://github.com/hashicorp/hcl). Some parts of the configuration
cannot be interpreted until we build and walk the graph, since they depend
on the outcome of other parts of the configuration, and so these parts of
the configuration remain represented as the low-level HCL types
[`hcl.Body`](https://pkg.go.dev/github.com/hashicorp/hcl/v2/#Body)
and
[`hcl.Expression`](https://pkg.go.dev/github.com/hashicorp/hcl/v2/#Expression),
allowing OpenTofu to interpret them at a more appropriate time.
## State Manager
A _state manager_ is responsible for storing and retrieving snapshots of the
[OpenTofu state](https://opentofu.org/docs/language/state/index.html)
for a particular workspace. Each manager is an implementation of
some combination of interfaces in
[the `statemgr` package](https://pkg.go.dev/github.com/opentofu/opentofu/internal/states/statemgr),
with most practical managers implementing the full set of operations
described by
[`statemgr.Full`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/states/statemgr#Full)
provided by a _backend_. The smaller interfaces exist primarily for use in
other function signatures to be explicit about what actions the function might
take on the state manager; there is little reason to write a state manager
that does not implement all of `statemgr.Full`.
The implementation
[`statemgr.Filesystem`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/states/statemgr#Filesystem) is used
by default (by the `local` backend) and is responsible for the familiar
`terraform.tfstate` local file that most OpenTofu users start with, before
they switch to [remote state](https://opentofu.org/docs/language/state/remote).
Other implementations of `statemgr.Full` are used to implement remote state.
Each of these saves and retrieves state via a remote network service
appropriate to the backend that creates it.
A state manager accepts and returns a state snapshot as a
[`states.State`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/states#State)
object. The state manager is responsible for exactly how that object is
serialized and stored, but all state managers at the time of writing use
the same JSON serialization format, storing the resulting JSON bytes in some
kind of arbitrary blob store.
## Graph Builder
A _graph builder_ is called by a
[`tofu.Context`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#Context)
method (e.g. `Plan` or `Apply`) to produce the graph that will be used
to represent the necessary steps for that operation and the dependency
relationships between them.
In most cases, the
[vertices](https://en.wikipedia.org/wiki/Vertex_(graph_theory)) of OpenTofu's
graphs each represent a specific object in the configuration, or something
derived from those configuration objects. For example, each `resource` block
in the configuration has one corresponding
[`GraphNodeConfigResource`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#GraphNodeConfigResource)
vertex representing it in the "plan" graph. (OpenTofu Core uses terminology
inconsistently, describing graph _vertices_ also as graph _nodes_ in various
places. These both describe the same concept.)
The [edges](https://en.wikipedia.org/wiki/Glossary_of_graph_theory_terms#edge)
in the graph represent "must happen after" relationships. These define the
order in which the vertices are evaluated, ensuring that e.g. one resource is
created before another resource that depends on it.
Each operation has its own graph builder, because the graph building process
is different for each. For example, a "plan" operation needs a graph built
directly from the configuration, but an "apply" operation instead builds its
graph from the set of changes described in the plan that is being applied.
The graph builders all work in terms of a sequence of _transforms_, which
are implementations of
[`tofu.GraphTransformer`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#GraphTransformer).
Implementations of this interface just take a graph and mutate it in any
way needed, and so the set of available transforms is quite varied. Some
important examples include:
* [`ConfigTransformer`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#ConfigTransformer),
which creates a graph vertex for each `resource` block in the configuration.
* [`StateTransformer`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#StateTransformer),
which creates a graph vertex for each resource instance currently tracked
in the state.
* [`ReferenceTransformer`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#ReferenceTransformer),
which analyses the configuration to find dependencies between resources and
other objects and creates any necessary "happens after" edges for these.
* [`ProviderTransformer`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#ProviderTransformer),
which associates each resource or resource instance with exactly one
provider configuration (implementing
[the inheritance rules](https://opentofu.org/docs/language/providers/))
and then creates "happens after" edges to ensure that the providers are
initialized before taking any actions with the resources that belong to
them.
There are many more different graph transforms, which can be discovered
by reading the source code for the different graph builders. Each graph
builder uses a different subset of these depending on the needs of the
operation that is being performed.
The result of graph building is a
[`tofu.Graph`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#Graph), which
can then be processed using a _graph walker_.
## Graph Walk
The process of walking the graph visits each vertex of that graph in a way
which respects the "happens after" edges in the graph. The walk algorithm
itself is implemented in
[the low-level `dag` package](https://pkg.go.dev/github.com/opentofu/opentofu/internal/dag#AcyclicGraph.Walk)
(where "DAG" is short for [_Directed Acyclic Graph_](https://en.wikipedia.org/wiki/Directed_acyclic_graph)), in
[`AcyclicGraph.Walk`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/dag#AcyclicGraph.Walk).
However, the "interesting" OpenTofu walk functionality is implemented in
[`tofu.ContextGraphWalker`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#ContextGraphWalker),
which implements a small set of higher-level operations that are performed
during the graph walk:
* `EnterPath` is called once for each module in the configuration, taking a
module address and returning a
[`tofu.EvalContext`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#EvalContext)
that tracks objects within that module. `tofu.Context` is the _global_
context for the entire operation, while `tofu.EvalContext` is a
context for processing within a single module, and is the primary means
by which the namespaces in each module are kept separate.
Each vertex in the graph is evaluated, in an order that guarantees that the
"happens after" edges will be respected. If possible, the graph walk algorithm
will evaluate multiple vertices concurrently. Vertex evaluation code must
therefore make careful use of concurrency primitives such as mutexes in order
to coordinate access to shared objects such as the `states.State` object.
In most cases, we use the helper wrapper
[`states.SyncState`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/states#SyncState)
to safely implement concurrent reads and writes from the shared state.
## Vertex Evaluation
The action taken for each vertex during the graph walk is called
_execution_. Execution runs a sequence of arbitrary actions that make sense
for a particular vertex type.
For example, evaluation of a vertex representing a resource instance during
a plan operation would include the following high-level steps:
* Retrieve the resource's associated provider from the `EvalContext`. This
should already be initialized earlier by the provider's own graph vertex,
due to the "happens after" edge between the resource node and the provider
node.
* Retrieve from the state the portion relevant to the specific resource
instance being evaluated.
* Evaluate the attribute expressions given for the resource in configuration.
This often involves retrieving the state of _other_ resource instances so
that their values can be copied or transformed into the current instance's
attributes, which is coordinated by the `EvalContext`.
* Pass the current instance state and the resource configuration to the
provider, asking the provider to produce an _instance diff_ representing the
differences between the state and the configuration.
* Save the instance diff as part of the plan that is being constructed by
this operation.
Each execution step for a vertex is an implementation of
[`tofu.Execute`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#GraphNodeExecutable.Execute).
As with graph transforms, the behavior of these implementations varies widely:
whereas graph transforms can take any action against the graph, an `Execute`
implementation can take any action against the `EvalContext`.
The implementation of `tofu.EvalContext` used in real processing
(as opposed to testing) is
[`tofu.BuiltinEvalContext`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#BuiltinEvalContext).
It provides coordinated access to plugins, the current state, and the current
plan via the `EvalContext` interface methods.
In order to be executed, a vertex must implement
[`tofu.GraphNodeExecutable`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#GraphNodeExecutable),
which has a single `Execute` method that handles. There are numerous `Execute`
implementations with different behaviors, but some prominent examples are:
* [NodePlannableResource.Execute](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#NodePlannableResourceInstance.Execute), which handles the `plan` operation.
* [`NodeApplyableResourceInstance.Execute`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#NodeApplyableResourceInstance.Execute), which handles the main `apply` operation.
* [`NodeDestroyResourceInstance.Execute`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#EvalWriteState), which handles the main `destroy` operation.
A vertex must complete successfully before the graph walk will begin evaluation
for other vertices that have "happens after" edges. Evaluation can fail with one
or more errors, in which case the graph walk is halted and the errors are
returned to the user.
### Expression Evaluation
An important part of vertex evaluation for most vertex types is evaluating
any expressions in the configuration block associated with the vertex. This
completes the processing of the portions of the configuration that were not
processed by the configuration loader.
The high-level process for expression evaluation is:
1. Analyze the configuration expressions to see which other objects they refer
to. For example, the expression `aws_instance.example[1]` refers to one of
the instances created by a `resource "aws_instance" "example"` block in
configuration. This analysis is performed by
[`lang.References`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/lang#References),
or more often one of the helper wrappers around it:
[`lang.ReferencesInBlock`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/lang#ReferencesInBlock)
or
[`lang.ReferencesInExpr`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/lang#ReferencesInExpr)
1. Retrieve from the state the data for the objects that are referred to and
create a lookup table of the values from these objects that the
HCL evaluation code can refer to.
1. Prepare the table of built-in functions so that HCL evaluation can refer to
them.
1. Ask HCL to evaluate each attribute's expression (a
[`hcl.Expression`](https://pkg.go.dev/github.com/hashicorp/hcl/v2/#Expression)
object) against the data and function lookup tables.
In practice, steps 2 through 4 are usually run all together using one
of the methods on [`lang.Scope`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/lang#Scope);
most commonly,
[`lang.EvalBlock`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/lang#Scope.EvalBlock)
or
[`lang.EvalExpr`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/lang#Scope.EvalExpr).
Expression evaluation produces a dynamic value represented as a
[`cty.Value`](https://pkg.go.dev/github.com/zclconf/go-cty/cty#Value).
This Go type represents values from the OpenTofu language and such values
are eventually passed to provider plugins.
### Sub-graphs
Some vertices have a special additional behavior that happens after their
evaluation steps are complete, where the vertex implementation is given
the opportunity to build another separate graph which will be walked as part
of the evaluation of the vertex.
The main example of this is when a `resource` block has the `count` argument
set. In that case, the plan graph initially contains one vertex for each
`resource` block, but that graph then _dynamically expands_ to have a sub-graph
containing one vertex for each instance requested by the count. That is, the
sub-graph of `aws_instance.example` might contain vertices for
`aws_instance.example[0]`, `aws_instance.example[1]`, etc. This is necessary
because the `count` argument may refer to other objects whose values are not
known when the main graph is constructed, but become known while evaluating
other vertices in the main graph.
This special behavior applies to vertex objects that implement
[`tofu.GraphNodeDynamicExpandable`](https://pkg.go.dev/github.com/opentofu/opentofu/internal/tofu#GraphNodeDynamicExpandable).
Such vertices have their own nested _graph builder_, _graph walk_,
and _vertex evaluation_ steps, with the same behaviors as described in these
sections for the main graph. The difference is in which graph transforms
are used to construct the graph and in which evaluation steps apply to the
nodes in that sub-graph.

Some files were not shown because too many files have changed in this diff Show More