Ops Wiki Upgrade Log: Refactoring, Document Locks, and Local Semantic Search
I’ve been away from Guangzhou for a while and haven’t been able to publish new posts regularly — life comes first. Recently, I returned to the Ops Wiki project I open-sourced earlier and made a number of upgrades. This post walks through what changed and serves as a record of the work.
Upgrade Timeline
I made a few adjustments in August to better support my day-to-day use. More recently, I decided to combine this personal knowledge base with my previously open-sourced brain-mixproject to build a local personal assistant. That led to another round of upgrades—and gave me a chance to revisit and organize some concepts along the way.
The upgrades fall into three phases, each with a different focus:
- Phase 1 (August 4): Make the codebase easier to maintain.
- Phase 2 (August 11–13):Make writing documents smoother and allow individual articles to be locked.
- Phase 3 (October 8): Make it possible to search by meaning as the knowledge base grows, and let colleagues use the interface in different languages.
Phase 1: Refactoring the Project Structure
In the June version, the backend was essentially flat: app.py, models.py, views.py, and crypto.pywere all crammed together. The front-end assets lived directly in the repository's static/ and templates/directories, and the database file wiki.sqlitehad even been committed to version control.
The app worked, but adding more features was becoming awkward. If I wanted to find the encryption or decryption logic for a particular route, I had to jump back and forth between several large files.
The Refactor
I moved the backend into a package at src/ops_wiki/, split into four subpackages by responsibility:
core/: application assembly and configurationmodels/: data modelsroutes/: HTTP blueprints, divided intoauth,views, andadminservices/: reusable capabilities such as encryption/decryption, search, email, and permission checks
I also moved all front-end assets into web/, physically separating them from the backend. Following the principle that personal data should stay local, I moved runtime artifacts—including the database, imported images, and logs—out of the project directory and into a local directory specified by _ASSET_ROOT. That directory is now covered by .gitignore. The repository contains code, not runtime data.
I also added wsgi.py as the WSGI entry point to configure logging and create the application. For easier operations, I wrote a one-command run.sh script with start, stop, restart, and statusactions. It centrally manages log and PID file locations and supports two modes: Flask's built-in server for development and Gunicorn for production.
A Rewritten Editor, Too
In the same set of commits, I replaced the third-party Toast UI editor with a custom split-view editor. The left pane displays the Markdown source, while the right pane renders a live HTML preview using marked. A draggable divider separates the panes; dragging it to either edge switches to full-screen source or full-screen preview. Scrolling is synchronized in both directions: scroll in the source pane and the preview follows proportionally, or vice versa.
I also added background auto-save to make the editor feel more familiar to users accustomed to Microsoft Office. When typing has stopped for one full minute, the document is saved silently. Every keystroke resets the timer, so auto-save won’t interrupt an active editing session. After a successful save, a green checkmark and a status message appear on the right side of the toolbar. Manual saves display Saved successfully; auto-saves display Auto-saved successfully. The message fades out after five seconds.
At this point, the tool finally met my own usability bar. It had become reasonably comfortable to use.
Phase 2: Better Editing and Password-Locked Articles
Mermaid Diagrams and KaTeX Formulas
As a technical professional, I often need flowcharts, sequence diagrams, and mathematical formulas in technical documentation. Ops Wiki didn’t support these out of the box, so I integrated both rendering capabilities locally.
To use Mermaid, put a fenced code block tagged as mermaid in the Markdown source. In the preview, the code is replaced with a vector diagram that scales to the width of its container instead of creating a horizontal scrollbar.
Implementation-wise, after markedfinishes parsing, a post-processing step finds the rendered pre > code.language-mermaidnodes, passes them to Mermaid to generate SVG, and replaces the code blocks with the diagrams. A 180 ms debounce prevents large diagrams from being redrawn repeatedly while the user types.
Formula rendering follows a different path.
KaTeX uses an extension to marked to handle $...$for inline formulas and $$...$$ for block formulas during parsing. The extension runs after code blocks and inline code have been handled, so dollar signs inside code don't conflict with formulas, and underscores or asterisks inside formulas aren't mistaken for Markdown emphasis syntax.
If a formula is invalid, throwOnError is set to false, and the formula falls back to plain text instead of breaking the entire preview. I added this behavior after discovering how frustrating it was for a malformed formula to make the whole preview unusable.
The Mermaid and KaTeX scripts, stylesheets, and font files are all stored under web/static/vendor/. The entire system works offline and does not depend on an external CDN.
Password-Locking Individual Articles
This was the most complete feature in this phase. The goal is straightforward: you can put a password lock on an article, and even the owner must enter the password each time they open it. I had used a similar feature in Yuque and found it genuinely useful, so I decided to add it here.
At the data layer, I added two fields to the nodetable: is_locked (a Boolean) and lock_password_hash (a password hash stored using Werkzeug's PBKDF2 implementation, never the plaintext password).
At the API layer, GET /api/nodes/:id returns empty content for a locked article, prompting the frontend to display a password overlay. Unlocking uses a separate endpoint, POST /api/nodes/:id/unlock. The server returns plaintext only if the password is correct, and that plaintext is valid only for the current response; subsequent requests still treat the article as locked.
A lock can be enabled or its password changed through PUT /api/nodes/:id, but only for nodes of type article. Setting is_locked: false actually removes the protection and clears the stored hash. Locked articles show a lock icon in the left-hand tree, while the owner sees a separate key icon.
One detail worth calling out: locked articles are excluded from the semantic-search index. Their chunks and vectors are skipped when new content is indexed, and previously indexed data is removed during a rebuild. This prevents locked content from leaking into search results as either text snippets or vector matches.
The unlock flow looks like this:
Temporarily unlocking an article and removing its lock are two different actions.Entering the password grants access for that request only. Refresh the page, or leave the article and return, and you’ll need to enter the password again. To remove the lock permanently, unlock the article first and then choose Manage Lock → Remove Lock in the toolbar. The lock-management controls are hidden behind the overlay while the article is locked, so users won’t see an illogical Remove Lockoption before unlocking it.
A Few Smaller Improvements
I added a context menu to the document tree. Right-clicking a node opens actions such as rename, delete, import, and bulk permission management; right-clicking empty space offers the option to create a root folder. When dragging nodes to move them, the app suppresses the immediate expand action for a node that was just right-clicked, preventing an accidental expansion.
There’s also a small Back to Top button, inspired by the kind of control commonly found on blogs. It appears after the editing area has been scrolled past a threshold, and clicking it smoothly scrolls back to the top. For an owner, scrolling the Markdown pane also takes the preview back to the top through scroll synchronization. For a read-only visitor, the preview is the only scrollable area, so the button acts on that pane directly.
Phase 3: Semantic Search and Internationalization
This phase introduced two major capabilities: semantic search and support for both English and Chinese.
Semantic Search
Why Run Search Locally?
As a team’s documentation grows, searching by title keywords is no longer enough. People often remember what a document says but forget its title. Semantic search addresses this by letting you describe what you’re looking for in natural language. The system matches meaning rather than only matching literal words.
Since the application already uses SQLite as its relational database, I added the sqlite-vecextension for vector search and stored the vectors in the same wiki.sqlitedatabase. There's no separate vector database service to run. The embedding model is a local Qwen3-Embedding-0.6B instance running on the CPU, so it doesn't use the GPU. The trade-off is a slower initial indexing run, but the benefits are no extra services, no external dependencies to host, and a straightforward deployment.
Get Zhenhui Yuan’s stories in your inbox
Join Medium for free to get updates from this writer.
In my brain-mix project, I used bge-large-zh-v1.5 for embeddings. I chose to try Qwen3-Embedding-0.6B here as a newer alternative, partly in preparation for the personal AI assistant I want to build later. The 0.6B model is a practical fit for CPU inference and can run whether or not a machine has a GPU. Its 1,024-dimensional vectors are sufficient for this use case, so there's no need to use a larger model just for the sake of it.
How the Content Is Chunked
Representing an entire long article with one vector would make retrieval too coarse-grained. Instead, the system splits each article into chunks that combine its title and body:
- The article title is prepended to every chunk and included in the embedding context.
- Paragraphs separated by blank lines are the smallest units. They are greedily accumulated until a chunk contains roughly 450 tokens.
- Adjacent chunks overlap by 15% to reduce the chance of losing information at a boundary.
- Paragraphs longer than 512 tokens are hard-split by token count.
- Markdown syntax for images and links is removed, leaving the text.
How the Vectors Are Generated
Embedding follows the official Qwen3-Embedding recommendation: use the hidden state of the final token as the pooled representation, then apply L2 normalization to obtain a 1,024-dimensional vector.
Two details are deliberate. First, the tokenizer uses left padding so the final sequence position always corresponds to the last valid token. Second, queries include the official retrieval instruction — Given a web search query, retrieve relevant passages...—while documents do not. This follows the usage convention expected by Qwen3-Embedding's training: queries and documents use different prefixes, which produces more reliable results.
The model is a lazily loaded, process-level singleton running on the CPU. In my tests, four inference threads provided the best throughput; adding more caused the threads to compete for CPU resources. A background thread warms up the model during startup and keeps it in memory, avoiding a noticeable delay when the first search arrives.
Keeping the Index in Sync with Edits
This was the most substantial and failure-prone part of the feature. Whenever a user saves an article, its vectors need to be updated. But embedding is CPU-intensive, so it shouldn’t block the save request.
I used a combination of a background thread, a deduplicating queue, and completion events. The save endpoint submits the node to the queue and returns immediately. A long-running background thread processes queued tasks one by one, re-chunks the entire article, regenerates its embeddings, and replaces the old vectors. If the same article is saved several times while waiting in the queue, only one pending job is kept. Once processing finishes, the system records an event. The frontend polls every four seconds and displays a toast such as “Embeddings generated for ‘Article Title’.”
Why not use WebSockets? The frontend is just Flask plus fetch, with no persistent connection infrastructure. Polling solves the problem without adding another dependency.
Events are isolated by user, so each page sees only the indexing results triggered by that user. A cursor ensures that the first poll starts from the current position rather than treating historical events as new and showing duplicate notifications.
In addition to automatic updates at runtime, I added scripts/build_vectors.pyfor initial indexing and manual rebuilds. It supports incremental indexing, full rebuilds (--rebuild), processing a single article (--node-id), and cleaning up stale data (--prune). It uses the same indexing logic as the runtime path, keeping both workflows consistent.
Filtering Search Results
The search endpoint accepts a natural-language query and returns article titles, matching snippets, and similarity scores. Retrieval uses K-nearest-neighbor search through sqlite-vec, but two filters are essential:
- Permissions:return only articles owned by the current user or explicitly shared with them. The permission check is included in the query.
- Locks:exclude articles where
is_locked = 1, consistent with the article-locking behavior described earlier.
The candidate pool is much larger than the number of results we intend to display — at least 200 candidates. Long articles can produce many chunks, and a small candidate pool might be dominated by a single article, leaving only one or two distinct nodes after deduplication. The system ultimately keeps only the most relevant chunk for each node. Since the vectors are normalized, the distance is converted to a similarity score using 1 - d²/2, then clamped to the range from 0 to 1 and displayed as a percentage.
The retrieval flow is:
The sidebar now has two search modes: Title and Semantic. Switch to semantic mode, enter a natural-language description, and click a result to jump to the corresponding item in the document tree and open it.
Internationalization
The system was originally Chinese-only. To make it easier for GitHub users around the world to use, I added complete Chinese and English support, with English as the default language.
On the frontend, I used the standard data-i18napproach. Static text in index.htmlis tagged with data-i18nkeys, while strings assembled in JavaScript go through I18n.t(). The two dictionaries live in i18n.js. The language selector is a segmented Chinese | Englishswitch, shown on both the login page and the top bar; the two controls stay in sync. The selection is stored locally in the browser using localStorage and a cookie. Since users must be able to change languages before logging in, language preferences are not synchronized at the account level.
The backend approach is a little more unusual. Making every route handler look up translated messages would require extensive changes and make it easy to miss a string. Instead, handlers continue to use Chinese source messages. A mapping table uses those original messages as message IDs, and a custom JSON provider translates the error and messagefields into English during response serialization. Parameterized messages, such as Unsupported file format: .zip, are handled with regular-expression rules. This keeps changes to business logic to a minimum: new messages only need entries in the dictionary. Email notifications don't use JSON, so their templates use t() to select the appropriate language separately.
To keep this mechanism reliable, I added scripts/check_i18n.py. It checks four things: unexpected Chinese literals in front-end JavaScript (excluding comments and developer logs), missing dictionary entries for data-i18nkeys used in HTML, consistency of keys between the Chinese and English dictionaries, and whether all user-facing messages passed to backend jsonify calls can be translated. Running it after adding or changing UI text catches most omissions.
Other Updates
I also updated the dependency list to include packages such as sqlite-vec, transformers, and torch; rewrote the README and added an English version; and consolidated embedding settings in config.py. These settings include the model path, device, precision, thread count, batch size, and chunking parameters, and can all be overridden with environment variables. On startup, the app automatically creates the vector tables and applies idempotent migrations.
Deployment and Operations
There are a few constraints worth understanding before deploying the updated version.
Semantic search must run in a single process. The embedding model occupies about 2.7 GB of resident memory, and every worker would load its own copy. In addition, asynchronous indexing-completion events are kept in process memory. With multiple workers, the save request and the polling request might reach different processes, and the frontend might never receive its toast notification. That’s why run.sh fixes the Gunicorn worker count at one. Do not increase it.
The model and vectors are local. The default embedding-model path points to a directory on my own machine. On another machine, set OPS_WIKI_EMBED_MODEL to the correct path. The vectors live in the same database as the application data, so no additional service is required.
Build the index during initial setup.After installing the dependencies, run scripts/build_vectors.pyonce for existing articles. Normal editing will maintain the index asynchronously afterwards. On my Intel Core i5 machine, inference averages about 0.55 chunks per second. For roughly 800 chunks, a full initial indexing run takes an estimated 20–30 minutes.
Existing databases are migrated automatically. At startup, the application checks for and adds the node.is_locked and node.lock_password_hashcolumns if needed. It also creates the four vector-related tables idempotently, so there is no need to run a manual migration.
Known Limitations
There are still a few things I haven’t finished:
- Search-result snippets are stored as plaintext in the
node_chunktable for display. If that's a concern, the system could store only vectors and decrypt the text when retrieving results, at the cost of slower searches. I haven't implemented that yet. - Password verification for locked articles has no failed-attempt limit, and the unlock endpoint is not rate-limited. Given that the application is intended for local or internal-network use, I’ve accepted this limitation for now.
- The database is still SQLite, which may experience write-lock conflicts under heavy concurrent writes. The connection timeout is set to five seconds. It’s suitable for a small team of a few dozen people; a larger deployment should consider PostgreSQL.
- The application has not been hardened for public-internet exposure. The README explicitly warns against exposing it directly to the public internet.
Final Thoughts
These days, everything seems to come with AI, but I think the right choice depends on the situation. AI-powered office tools can certainly improve productivity, but they can also send a great deal of personal information and data outside your environment, unless you deploy them privately. Vendors repeatedly promise that personal data won’t be used for model training and that data collection and processing will comply with applicable laws and regulations. Still, there is no such thing as a free lunch: commercial services are driven by business incentives. I don’t want to assume the worst of anyone, but sometimes the most expensive things are the ones that appear to be free. I suspect data will increasingly be one of those things — especially clean, well-structured, high-quality data.
Protecting my own data is therefore one of the reasons I built this project.
I really admire LibreOffice for its willingness to say no to AI integration in an era when AI is being added to everything. I’m taking a similar approach: I’ll try to keep Ops Wiki lightweight and avoid depending on external services. I’ll continue updating it from time to time. Mobile support is still missing, but that should be manageable in a future update.
Project repository: https://github.com/yzh0623/ops-wiki
