On-Device WA Law Model
Your Data Your Privacy
US-State-Law-EmbeddingGemma-2-GGUF
A Q4_K_M quantized version of our fine-tuned legal embedding model, designed to make semantic retrieval more practical on consumer hardware, CPU-based systems, and resource-constrained edge devices.
Interactive concept demos · No model download required · No API key required
The 6× figure refers to vector storage at 128 dimensions versus 768 at equal precision. It is not a claim about model-file compression or measured inference speed.
A legal research tool in your pocket.
Imagine searching a collection of state laws directly from a mobile application. A local embedding model turns a question into a vector; a vector index retrieves relevant passages. Explore the simulated experience below.
Designed for local applications
A compatible runtime could generate query embeddings on the device. The app can then compare them with a precomputed local index, subject to available memory, compute, and runtime support.
What happens behind the search?
The embedding model represents the question as a vector.
A vector search engine ranks precomputed document embeddings.
The application displays matching text and source references.
Retrieval finds potentially relevant passages. A real application still needs verified legal sources, correct citations, jurisdiction checks, and appropriate legal interpretation.
Find the moment. Not just the file.
Long recordings can contain hours of potentially relevant material. With timestamped transcripts or video descriptions embedded and indexed, investigators could search for concepts in natural language and navigate to candidate moments instead of reviewing every second manually.
Video Search Explorer
1. Transcribe or describe
Generate timestamped speech transcripts and/or descriptions of selected video frames or clips.
2. Embed and index
Encode text segments and store their vectors alongside timestamps and source metadata.
3. Retrieve moments
Search for concepts, review candidate matches, and jump to relevant timestamps.
Smaller vectors. More room to search.
Matryoshka Representation Learning (MRL) supports using shorter prefixes of the model's embedding vector. Choose a dimension below to explore how the storage footprint changes relative to a 768-dimensional vector.
Embedding dimension
Interactive explorerConceptual illustration only; individual vector values are not shown.
The full 768-dimensional representation is the baseline for comparison.
What does this mean?
Smaller embeddings can reduce vector-index storage and memory requirements. Retrieval quality may change with dimensionality, so evaluate the chosen dimension on your own legal corpus and task.
| Dimensions | Relative storage | Reduction | Storage multiplier |
|---|---|---|---|
| 768 | 100% | 0% | 1× |
| 512 | 66.7% | 33.3% | 1.5× smaller |
| 256 | 33.3% | 66.7% | 3× smaller |
| 128 | 16.7% | 83.3% | 6× smaller |
Calculations assume the same numeric precision and storage representation per dimension. They describe vector payloads, not total model size, runtime memory, or complete database size. Use these dimensions only if the actual model and inference pipeline support them.
Three pieces. Different jobs.
Quantization, embedding dimensionality, and vector retrieval address different parts of a local search system.
GGUF + Q4_K_M
GGUF is a model-file format used by supported local inference runtimes. Q4_K_M is a 4-bit-oriented quantization scheme that reduces weight storage relative to higher-precision representations.
Model footprintMRL dimensions
Supported truncated embeddings can reduce the space required to store document vectors. This affects the retrieval index rather than the quantized model's weight file.
Vector footprintVector retrieval
A vector index compares query embeddings with document embeddings and returns ranked candidates. The index, corpus, and retrieval logic are separate components of the application.
Search pipelineTwo possible deployment architectures
The GGUF embedding model does not itself transcribe audio, understand raw video, or determine whether an action is lawful. Those capabilities require separate components. Any stage can run locally only when its software and hardware support it.
Build your own local retrieval pipeline.
A production implementation needs a compatible embedding runtime, correct task formatting, document preprocessing, and an index. Verify the exact GGUF artifact and runtime before deploying.
Runtime compatibility
Check whether your selected runtime supports embedding tasks, the model architecture, pooling, required prompts, and the intended output dimensions.
Explore llama.cppCompatibility is not guaranteed merely because a runtime supports GGUF.
Before running the model
Confirm the exported artifact preserves the correct embedding behavior.
import torch from sentence_transformers import SentenceTransformer, util # 1. Load the fine-tuned model model = SentenceTransformer('CSI-lab/US-state-law-EmbeddingGemma-2', model_kwargs={ "torch_dtype": torch.bfloat16, "config_kwargs": { "load_vision_encoder": False, "load_audio_encoder": False } }) # 2. Define the laws (Your Vector Database) laws = [ "RCW 9A.56.030: Theft in the first degree. A person is guilty of theft in the first degree if he or she commits theft of property or services which exceed(s) five thousand dollars in value.", "RCW 46.61.502: Driving under the influence. A person is guilty of driving while under the influence of intoxicating liquor...", "RCW 9A.36.011: Assault in the first degree. A person is guilty of assault in the first degree if he or she..." ] # 3. Define the user's search query user_query = "What dollar amount makes a theft a first degree felony?" # 4. Encode the documents and the query query_embedding = model.encode(user_query, convert_to_tensor=True, prompt_name="SearchQuery") law_embeddings = model.encode(laws, convert_to_tensor=True, prompt_name="Document") # 5. Calculate Cosine Similarity cosine_scores = util.cos_sim(query_embedding, law_embeddings) # 7. Print the top result best_idx = cosine_scores.argmax().item() print(f"Top Match: {laws[best_idx]}") print(f"Similarity Score: {cosine_scores[0][best_idx]:.4f}")