docs(05-shell-decoder): create phase plan

2026-02-25 01:33:31 +03:00
parent 79a7ce2010
commit 7331f4c0bb
3 changed files with 447 additions and 3 deletions
--- a/.planning/ROADMAP.md
+++ b/.planning/ROADMAP.md
@@ -88,10 +88,11 @@ Plans:
  1. Shell script extracts all files from an archive created by the Rust archiver, producing byte-identical output
  2. Script uses only standard busybox commands: dd, xxd, openssl (with -K/-iv/-nosalt for raw key mode), gunzip
  3. Script correctly handles files with non-ASCII names (Cyrillic characters)
-**Plans**: TBD
+**Plans**: 2 plans

 Plans:
- [ ] 05-01: TBD
+- [ ] 05-01-PLAN.md -- Shell decode.sh: busybox-compatible decoder with full pipeline (dd/xxd/openssl/gunzip)
+- [ ] 05-02-PLAN.md -- Cross-validation test script and end-to-end verification (6 test cases)

 ### Phase 6: Obfuscation Hardening
 **Goal**: Archive format resists casual analysis -- binwalk, file, strings, and hex editors reveal nothing useful
@@ -118,5 +119,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6
 | 2. Core Archiver | 2/2 | Complete    | 2026-02-24 |
 | 3. Round-Trip Verification | 2/2 | Complete    | 2026-02-24 |
 | 4. Kotlin Decoder | 1/1 | Complete    | 2026-02-24 |
-| 5. Shell Decoder | 0/1 | Not started | - |
+| 5. Shell Decoder | 0/2 | Not started | - |
 | 6. Obfuscation Hardening | 0/1 | Not started | - |
--- a/.planning/phases/05-shell-decoder/05-01-PLAN.md
+++ b/.planning/phases/05-shell-decoder/05-01-PLAN.md
@@ -0,0 +1,230 @@
+---
+phase: 05-shell-decoder
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - shell/decode.sh
+autonomous: true
+requirements:
+  - SHL-01
+  - SHL-02
+  - SHL-03
+
+must_haves:
+  truths:
+    - "Shell script extracts all files from a Rust-created archive, byte-identical to originals"
+    - "Script uses only dd, xxd/od, openssl, gunzip, sha256sum -- no bash-specific syntax"
+    - "Script decrypts files using openssl enc -aes-256-cbc with raw hex key (-K/-iv/-nosalt)"
+    - "Script correctly handles files with Cyrillic UTF-8 names"
+    - "Script verifies HMAC-SHA-256 before decryption (graceful degradation if openssl lacks HMAC support)"
+    - "Script verifies SHA-256 after decompression"
+  artifacts:
+    - path: "shell/decode.sh"
+      provides: "Busybox-compatible archive decoder shell script"
+      min_lines: 150
+      contains: "openssl enc -d -aes-256-cbc"
+  key_links:
+    - from: "shell/decode.sh"
+      to: "docs/FORMAT.md Section 13"
+      via: "read_hex, read_le_u16, read_le_u32 functions from spec"
+      pattern: "read_le_u32|read_le_u16|read_hex"
+    - from: "shell/decode.sh"
+      to: "src/key.rs"
+      via: "Hardcoded KEY_HEX constant matching Rust key bytes"
+      pattern: "7a35c1d94fe82b6a910df358bc74a61e428fd063e5179b2cfa8406cd3e79b550"
+    - from: "shell/decode.sh"
+      to: "openssl enc"
+      via: "AES-256-CBC decryption with raw key mode"
+      pattern: "openssl enc -d -aes-256-cbc -nosalt -K.*-iv"
+---
+
+<objective>
+Create the busybox-compatible shell decoder script that extracts files from archives created by the Rust archiver.
+
+Purpose: Provide a fallback extraction path when Kotlin/Android is unavailable. The script must work on minimal busybox systems with only dd, xxd/od, openssl, gunzip, and sha256sum.
+
+Output: `shell/decode.sh` -- a single self-contained POSIX shell script implementing the full decode pipeline from FORMAT.md Section 10.
+</objective>
+
+<execution_context>
+@/home/nick/.claude/get-shit-done/workflows/execute-plan.md
+@/home/nick/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@docs/FORMAT.md
+@src/key.rs
+@.planning/phases/05-shell-decoder/05-RESEARCH.md
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Create shell/decode.sh with full decode pipeline</name>
+  <files>shell/decode.sh</files>
+  <action>
+Create `shell/decode.sh` -- a single self-contained POSIX shell script that decodes archives created by the Rust archiver. The script MUST be compatible with busybox ash/sh (NO bash-specific syntax: no `[[ ]]`, no arrays, no `$((16#FF))`, no process substitution `<()`).
+
+**Script structure (follow this order):**
+
+1. **Shebang and usage:**
+   - `#!/bin/sh` (NOT `#!/bin/bash`)
+   - Usage: `decode.sh <archive_file> <output_dir>`
+   - Validate exactly 2 arguments; print usage and exit 1 otherwise
+   - Create output directory if it doesn't exist: `mkdir -p "$OUTPUT_DIR"`
+
+2. **Hardcoded key constant:**
+   ```sh
+   KEY_HEX="7a35c1d94fe82b6a910df358bc74a61e428fd063e5179b2cfa8406cd3e79b550"
+   ```
+   Verify this matches src/key.rs bytes: 0x7A 0x35 0xC1 0xD9 0x4F 0xE8 0x2B 0x6A 0x91 0x0D 0xF3 0x58 0xBC 0x74 0xA6 0x1E 0x42 0x8F 0xD0 0x63 0xE5 0x17 0x9B 0x2C 0xFA 0x84 0x06 0xCD 0x3E 0x79 0xB5 0x50
+
+3. **Prerequisite checks:**
+   - Check for `dd`, `openssl`, `sha256sum` -- exit with error if missing
+   - Do NOT require `gunzip` at startup (only needed if compressed files exist)
+   - Detect `xxd` vs `od` fallback using `command -v xxd`
+
+4. **Temporary directory with cleanup trap:**
+   ```sh
+   TMPDIR=$(mktemp -d)
+   trap 'rm -rf "$TMPDIR"' EXIT
+   ```
+
+5. **Hex reading functions (from FORMAT.md Section 13.1 + 13.2):**
+   - `read_hex "$file" "$offset" "$count"` -- returns lowercase hex string
+   - If xxd available: `dd ... | xxd -p | tr -d '\n'`
+   - If xxd NOT available: `dd ... | od -A n -t x1 | tr -d ' \n'`
+   - `read_le_u16 "$file" "$offset"` -- reads 2 bytes LE, prints decimal
+   - `read_le_u32 "$file" "$offset"` -- reads 4 bytes LE, prints decimal
+   - Use `printf '%d' "0x${swapped_hex}"` for hex-to-decimal conversion (POSIX compatible)
+
+6. **HMAC availability detection (from FORMAT.md Section 13.3):**
+   ```sh
+   SKIP_HMAC=0
+   if ! echo -n "test" | openssl dgst -sha256 -mac HMAC -macopt hexkey:00 >/dev/null 2>&1; then
+     echo "WARNING: openssl HMAC not supported, skipping HMAC verification"
+     SKIP_HMAC=1
+   fi
+   ```
+
+7. **Header parsing (FORMAT.md Section 4):**
+   - Read magic bytes at offset 0, count 4. Verify equals "00ea7263" (lowercase hex). If not, error "Invalid archive: bad magic bytes"
+   - Read version at offset 4. Verify equals 1. If not, error "Unsupported version"
+   - Read flags at offset 5 (1 byte)
+   - Read file_count at offset 6 (u16 LE)
+   - Read toc_offset at offset 8 (u32 LE)
+   - Read toc_size at offset 12 (u32 LE)
+   - Print: "Archive: N files"
+
+8. **TOC parsing loop (FORMAT.md Section 5):**
+   - Start at `pos=$toc_offset`
+   - Loop `i` from 0 to `$((file_count - 1))` using a while loop (not seq, for POSIX compat; actually `seq` IS available in busybox so either is fine)
+   - For each entry, parse sequentially:
+     - name_length (u16 LE at pos), advance pos by 2
+     - filename: `dd if="$ARCHIVE" bs=1 skip="$pos" count="$name_length" 2>/dev/null` -- raw UTF-8 bytes go directly into variable (handles Cyrillic per SHL-03)
+     - advance pos by name_length
+     - original_size (u32 LE), advance pos by 4
+     - compressed_size (u32 LE), advance pos by 4
+     - encrypted_size (u32 LE), advance pos by 4
+     - data_offset (u32 LE), advance pos by 4
+     - iv_hex: `read_hex "$ARCHIVE" "$pos" 16`, advance pos by 16
+     - hmac_hex: `read_hex "$ARCHIVE" "$pos" 32`, advance pos by 32
+     - sha256_hex: `read_hex "$ARCHIVE" "$pos" 32`, advance pos by 32
+     - compression_flag: `read_hex "$ARCHIVE" "$pos" 1`, advance pos by 1
+     - padding_after (u16 LE), advance pos by 2
+   - Store iv_toc_offset separately (the offset where IV was read) for HMAC verification
+
+9. **Per-file decode pipeline (FORMAT.md Section 10):**
+   For each file in the TOC loop:
+
+   a. **Extract ciphertext to temp file:**
+      ```sh
+      dd if="$ARCHIVE" bs=1 skip="$data_offset" count="$encrypted_size" of="$TMPDIR/ct.bin" 2>/dev/null
+      ```
+
+   b. **Verify HMAC (if available):**
+      - Only if `SKIP_HMAC=0`
+      - HMAC input = IV bytes (from archive, NOT from hex variable) || ciphertext bytes
+      - Extract IV bytes from archive at the iv_toc_offset position (16 bytes) and cat with ciphertext:
+        ```sh
+        computed_hmac=$( { dd if="$ARCHIVE" bs=1 skip="$iv_toc_pos" count=16 2>/dev/null; cat "$TMPDIR/ct.bin"; } | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${KEY_HEX}" -hex 2>/dev/null | awk '{print $NF}' )
+        ```
+      - Normalize both to lowercase: `echo "$hex" | tr 'A-F' 'a-f'`
+      - If mismatch: print "HMAC FAILED for <filename>, skipping" to stderr and `continue` to next file
+
+   c. **Decrypt:**
+      ```sh
+      openssl enc -d -aes-256-cbc -nosalt -K "$KEY_HEX" -iv "$iv_hex" -in "$TMPDIR/ct.bin" -out "$TMPDIR/dec.bin"
+      ```
+
+   d. **Decompress (if compression_flag = "01"):**
+      ```sh
+      if [ "$compression_flag" = "01" ]; then
+        gunzip -c "$TMPDIR/dec.bin" > "$TMPDIR/out.bin"
+      else
+        mv "$TMPDIR/dec.bin" "$TMPDIR/out.bin"
+      fi
+      ```
+      Handle special case: if original_size is 0, create empty file directly (`touch "$TMPDIR/out.bin"`)
+
+   e. **Verify SHA-256:**
+      ```sh
+      actual_sha=$(sha256sum "$TMPDIR/out.bin" | awk '{print $1}')
+      ```
+      If mismatch: print "WARNING: SHA-256 mismatch for <filename>" to stderr (but still write the file, matching Rust/Kotlin behavior)
+
+   f. **Write output file:**
+      ```sh
+      mv "$TMPDIR/out.bin" "$OUTPUT_DIR/$filename"
+      ```
+      Print progress: "Extracted: <filename> (<original_size> bytes)"
+
+10. **Final summary:**
+    Print "Done: extracted N files to <output_dir>"
+
+**CRITICAL anti-patterns to avoid:**
+- NO `[[ ]]` -- use `[ ]` only
+- NO bash arrays
+- NO `$((16#FF))` -- use `printf '%d' "0x..."` instead
+- NO process substitution `<()`
+- NO `echo -e` -- use `printf` for anything requiring escape sequences
+- ALL `dd` commands must have `2>/dev/null`
+- Set `export LC_ALL=C` near the top (for predictable byte handling)
+
+Make the script executable: `chmod +x shell/decode.sh`
+  </action>
+  <verify>
+    <automated>cd /home/nick/Projects/Rust/encrypted_archive && test -f shell/decode.sh && test -x shell/decode.sh && sh -n shell/decode.sh && echo "SYNTAX OK" && grep -q 'openssl enc -d -aes-256-cbc' shell/decode.sh && grep -q '7a35c1d94fe82b6a910df358bc74a61e428fd063e5179b2cfa8406cd3e79b550' shell/decode.sh && echo "KEY OK" && ! grep -E '\[\[|BASH_SOURCE|\$\(\(16#' shell/decode.sh && echo "NO BASH-ISMS"</automated>
+    <manual>Review shell/decode.sh for POSIX compliance, correct FORMAT.md field offsets, and complete pipeline</manual>
+  </verify>
+  <done>shell/decode.sh exists, is executable, passes sh -n syntax check, contains the correct KEY_HEX, uses openssl enc -d -aes-256-cbc, and has no bash-specific syntax</done>
+</task>
+
+</tasks>
+
+<verification>
+1. `sh -n shell/decode.sh` passes (valid POSIX shell syntax)
+2. Script contains correct hardcoded key matching src/key.rs
+3. Script contains openssl enc -d -aes-256-cbc -nosalt -K -iv invocation
+4. Script has xxd/od fallback detection
+5. Script has HMAC graceful degradation
+6. No bash-isms: no `[[`, no `BASH_SOURCE`, no `$((16#...))`, no arrays
+</verification>
+
+<success_criteria>
+- shell/decode.sh is a valid POSIX shell script (passes sh -n)
+- Script implements complete decode pipeline: header parse -> TOC parse -> HMAC verify -> decrypt -> decompress -> SHA-256 verify -> write
+- Hardcoded key matches src/key.rs
+- xxd/od fallback for hex conversion
+- Graceful HMAC degradation
+- UTF-8 filename preservation for Cyrillic names
+</success_criteria>
+
+<output>
+After completion, create `.planning/phases/05-shell-decoder/05-01-SUMMARY.md`
+</output>
--- a/.planning/phases/05-shell-decoder/05-02-PLAN.md
+++ b/.planning/phases/05-shell-decoder/05-02-PLAN.md
@@ -0,0 +1,213 @@
+---
+phase: 05-shell-decoder
+plan: 02
+type: execute
+wave: 2
+depends_on:
+  - "05-01"
+files_modified:
+  - shell/test_decoder.sh
+autonomous: true
+requirements:
+  - SHL-01
+  - SHL-02
+  - SHL-03
+
+must_haves:
+  truths:
+    - "Shell decoder extracts single text file byte-identical to original"
+    - "Shell decoder extracts multiple files (text + binary) byte-identical to originals"
+    - "Shell decoder extracts files with Cyrillic UTF-8 filenames correctly"
+    - "Shell decoder handles no-compress mode (raw encrypted, not gzip-compressed)"
+    - "Shell decoder handles empty (0-byte) files"
+    - "Shell decoder handles large files (100 KB+)"
+  artifacts:
+    - path: "shell/test_decoder.sh"
+      provides: "Cross-validation test script: Rust pack -> Shell decode -> SHA-256 verify"
+      min_lines: 150
+      contains: "sha256sum"
+  key_links:
+    - from: "shell/test_decoder.sh"
+      to: "shell/decode.sh"
+      via: "Invokes decode.sh to decode Rust-created archives"
+      pattern: "decode\\.sh"
+    - from: "shell/test_decoder.sh"
+      to: "target/release/encrypted_archive"
+      via: "Uses Rust archiver to create test archives"
+      pattern: "encrypted_archive.*pack"
+---
+
+<objective>
+Create cross-validation test script that verifies the shell decoder produces byte-identical output to originals for all edge cases.
+
+Purpose: Prove the shell decoder correctly implements the archive format by testing against Rust-created archives (same pattern as kotlin/test_decoder.sh).
+
+Output: `shell/test_decoder.sh` -- a test script that creates archives with the Rust CLI, decodes with decode.sh, and verifies byte-identical output via SHA-256 comparison.
+</objective>
+
+<execution_context>
+@/home/nick/.claude/get-shit-done/workflows/execute-plan.md
+@/home/nick/.claude/get-shit-done/templates/summary.md
+</execution_context>
+
+<context>
+@.planning/PROJECT.md
+@.planning/ROADMAP.md
+@.planning/STATE.md
+@.planning/phases/05-shell-decoder/05-01-SUMMARY.md
+@kotlin/test_decoder.sh
+@shell/decode.sh
+</context>
+
+<tasks>
+
+<task type="auto">
+  <name>Task 1: Create shell/test_decoder.sh cross-validation test script</name>
+  <files>shell/test_decoder.sh</files>
+  <action>
+Create `shell/test_decoder.sh` modeled after `kotlin/test_decoder.sh` but adapted for the shell decoder. This script uses bash (it runs on the development machine, not the target device) but invokes `decode.sh` with `sh` to ensure POSIX compatibility.
+
+**Script structure (follow kotlin/test_decoder.sh pattern):**
+
+1. **Header:**
+   - `#!/usr/bin/env bash`
+   - `set -euo pipefail`
+   - Comment: Cross-validation test script for Shell decoder
+
+2. **Variables and helpers (copy pattern from kotlin/test_decoder.sh):**
+   - SCRIPT_DIR, PROJECT_DIR, TMPDIR, PASS_COUNT, FAIL_COUNT, TOTAL_COUNT
+   - Colors (GREEN, RED, YELLOW, BOLD, NC) with terminal detection
+   - `cleanup()` function with trap
+   - `pass()`, `fail()`, `sha256_of()`, `verify_file()` helper functions (identical to kotlin version)
+
+3. **Prerequisites check:**
+   - Check for `cargo` (to build Rust archiver)
+   - Check for `openssl` (needed by decode.sh)
+   - Check for `sh` (always available, but verify)
+   - Print versions: cargo, openssl, sh
+
+4. **Build Rust archiver:**
+   ```bash
+   (cd "$PROJECT_DIR" && cargo build --release -q)
+   ARCHIVER="$PROJECT_DIR/target/release/encrypted_archive"
+   ```
+
+5. **Decoder path:**
+   ```bash
+   DECODER="$SCRIPT_DIR/decode.sh"
+   ```
+   Verify it exists and is executable.
+
+6. **Test cases (6 tests matching kotlin/test_decoder.sh):**
+
+   **Test 1: Single text file**
+   - Create `hello.txt` with content "Hello, World!" (no newline: `echo -n`)
+   - Pack with Rust archiver
+   - Decode with: `sh "$DECODER" "$TMPDIR/test1.archive" "$TMPDIR/output1/"`
+   - verify_file hello.txt
+
+   **Test 2: Multiple files (text + binary)**
+   - Create text.txt with 3 lines
+   - Create binary.bin with 10 KB random data (`dd if=/dev/urandom bs=1024 count=10`)
+   - Pack both files
+   - Decode with sh
+   - verify_file text.txt, binary.bin
+
+   **Test 3: No-compress mode**
+   - Create fake.dat with 2.5 KB random data
+   - Pack with `--no-compress "fake.dat"`
+   - Decode with sh
+   - verify_file fake.dat
+
+   **Test 4: Empty file**
+   - Create empty.txt with `touch`
+   - Pack with Rust archiver
+   - Decode with sh
+   - Verify output file exists and is 0 bytes (same logic as kotlin test)
+
+   **Test 5: Large file (100 KB)**
+   - Create large.bin with 100 KB random data
+   - Pack with Rust archiver
+   - Decode with sh
+   - verify_file large.bin
+
+   **Test 6: Cyrillic UTF-8 filename (SHL-03 validation)**
+   - Create a file with a Cyrillic name: `echo -n "Тестовое содержимое" > "$TMPDIR/файл.txt"`
+   - Pack with Rust archiver
+   - Decode with sh
+   - verify_file that the Cyrillic-named file exists and matches SHA-256
+   - This test specifically validates SHL-03
+
+7. **Summary:**
+   - Print results: "N passed, M failed out of T tests"
+   - Exit 1 if any failures, exit 0 if all pass
+
+**Key difference from kotlin test:** Instead of `java -jar "$JAR"`, use `sh "$DECODER"` to invoke the shell decoder. This ensures the decoder is tested under POSIX sh, not bash.
+
+Make the script executable: `chmod +x shell/test_decoder.sh`
+  </action>
+  <verify>
+    <automated>cd /home/nick/Projects/Rust/encrypted_archive && test -f shell/test_decoder.sh && test -x shell/test_decoder.sh && bash -n shell/test_decoder.sh && echo "SYNTAX OK" && grep -q 'decode.sh' shell/test_decoder.sh && grep -q 'verify_file' shell/test_decoder.sh && echo "STRUCTURE OK"</automated>
+    <manual>Run `bash shell/test_decoder.sh` and verify all 6 tests pass</manual>
+  </verify>
+  <done>shell/test_decoder.sh exists, is executable, passes bash -n, and contains all 6 test cases including Cyrillic filename test</done>
+</task>
+
+<task type="auto">
+  <name>Task 2: Run cross-validation tests and fix any decode.sh issues</name>
+  <files>shell/decode.sh, shell/test_decoder.sh</files>
+  <action>
+Run the cross-validation test script and fix any issues discovered:
+
+```bash
+cd /home/nick/Projects/Rust/encrypted_archive
+bash shell/test_decoder.sh
+```
+
+**Expected:** All 6 tests pass (PASS for each test case).
+
+**If tests fail:**
+- Read the error output carefully
+- Common issues to look for:
+  1. **Hex case mismatch:** HMAC or SHA-256 comparison fails because of uppercase vs lowercase hex. Fix: normalize with `tr 'A-F' 'a-f'`
+  2. **dd offset errors:** Wrong offset calculation in TOC parsing. Fix: verify field order matches FORMAT.md Section 5 exactly
+  3. **openssl pipe issues:** Decryption produces garbage. Fix: extract ciphertext to temp file first, then decrypt from file (not pipe)
+  4. **Empty file handling:** gunzip fails on empty input. Fix: check original_size=0 before decompression
+  5. **Cyrillic filename:** garbled characters. Fix: ensure LC_ALL=C and no text processing on filename bytes
+  6. **HMAC scope error:** Wrong bytes fed to HMAC. Fix: HMAC = IV (from archive file, 16 bytes) || ciphertext (encrypted_size bytes)
+
+Fix decode.sh until all tests pass. Do NOT modify the test script to make tests pass -- fix the decoder.
+
+**After all tests pass**, run one more verification:
+```bash
+# Verify Rust round-trip still works
+cd /home/nick/Projects/Rust/encrypted_archive && cargo test --release 2>&1 | tail -5
+```
+  </action>
+  <verify>
+    <automated>cd /home/nick/Projects/Rust/encrypted_archive && bash shell/test_decoder.sh 2>&1 | tail -20 && echo "---" && cargo test --release 2>&1 | tail -3</automated>
+    <manual>All 6 shell decoder tests pass, including Cyrillic filename. Rust tests still pass.</manual>
+  </verify>
+  <done>All 6 cross-validation tests pass (single file, multiple files, no-compress, empty file, large file, Cyrillic filename). Existing Rust tests still pass.</done>
+</task>
+
+</tasks>
+
+<verification>
+1. `bash shell/test_decoder.sh` runs all 6 tests and reports "ALL TESTS PASSED"
+2. Test 6 specifically validates Cyrillic filename extraction (SHL-03)
+3. Test 3 validates no-compress mode with openssl raw decryption (SHL-02)
+4. All tests use Rust archiver to pack and shell decoder to unpack (SHL-01)
+5. `cargo test --release` still passes (no regressions)
+</verification>
+
+<success_criteria>
+- shell/test_decoder.sh exists and is executable
+- All 6 test cases pass: single file, multiple files, no-compress, empty file, large file, Cyrillic filename
+- Shell decoder produces byte-identical output verified by SHA-256 comparison
+- Existing Rust test suite still passes
+</success_criteria>
+
+<output>
+After completion, create `.planning/phases/05-shell-decoder/05-02-SUMMARY.md`
+</output>