Skip to content

feat: Implement GeometricSLM ( replace legacy lqa task )#17

Open
Concode0 wants to merge 3 commits into
mainfrom
feat/slm-geometric
Open

feat: Implement GeometricSLM ( replace legacy lqa task )#17
Concode0 wants to merge 3 commits into
mainfrom
feat/slm-geometric

Conversation

@Concode0
Copy link
Copy Markdown
Owner

@Concode0 Concode0 commented May 8, 2026

This PR introduces the foundational implementation of a Geometric Small Language Model (SLM) using the Versor framework, replacing the existing LQA task. It demonstrates the viability and potential of geometric language modeling.

  • Stable Training: Confirmed stable gradient convergence without dimension-induced memory explosion.
  • Metrics: Achieved an Eval PPL of 384.92 and a Token Accuracy of 12.97% on a Cosmopedia subset.

Environment Notes & Known Issues
This initial implementation was developed and verified on an M4 MacBook Air. Please be aware of the following hardware constraints:

  • MPS Instability: torch.linalg operations and torch.compile are currently unstable on Apple Silicon (MPS) backends.
  • Recommended Setup: We highly recommend running this in eager mode or, preferably, in a native CUDA environment.
  • Memory Management: Due to the high memory consumption of geometric operations, you may need to manually tune ATTN_BLOCK_SIZE according to your available VRAM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant