Texo’s Menagerie of Distractions

This is where I dump progress reports on projects and ideas I’ve been working on but haven’t fully explored yet. Think of it as my public to-do list of things I want to get back to someday.

For a slightly less chaotic presentation of my work, go here.

---

• Parserator - my parser combinators library

  • I’ve been exploring the Haskell library Megaparsec for some time now, mostly to find features that might be useful for Parserator. I think I could borrow a lot of ideas, especially when it comes to error handling and recovery.
  • Some improvements I’ve been thinking about:
    • Packrat parsing + backtracking with memoization - Memoizes parse results to avoid redundant parsing of the same input, significantly improving performance for complex grammars
    • Static analysis to examine parser structure and find optimization opportunities
    • Hotspot optimization focusing on the performance-critical parts (need to add profiling first)
    • Better character class handling using bitwise operations for speed
    • Streaming and buffering improvements to handle large inputs without choking memory
  • I love Typescript and want to make it a great choice for prototyping new programming languages.

• TypeLisp - a Lisp written entirely in TypeScript Types [screenshot]

  • Borrowed a lot of ideas from David Blass’s work on ArkType. He uses a type-level shift-reduce parser which turns out to be a whole lot less taxing on the typechecker than some of the previous type-level parsers I’d written, like this one for a weird xml-like language which I wrote as a tongue-in-cheek response on twitter.
  • I still have quite a lot more I’d like to implement, like user-defined functions and macros. I also ought to use a better arithmetic encoding that can actually scale to numbers beyond 999.

• A language designed for reactive notebooks

  • I got a basic reactive notebook for Typescript working here. Here’s a screenshot. I’d used Observable notebooks (I even got paid for them!) quite a bit in the past, and really struggled to keep it together when working on large notebooks because of the absence of types.
  • This was the exact moment I realized that it might be fun to try and build a language exclusively to be used in reactive notebooks, and this rather interesting thread ensued.
    • Discovered Eve through this thread, fascinating. Someone in the thread said

      discovering Eve for me was like finding the abandoned relics of an advanced alien civilization

    • Also discovered Hazel through this thread (Someone that works/worked on it popped up in the thread). I don’t think I’ve been as inspired by something in a long time.

• Prisma to Effect Schema Codegen

  • Used my parsing library, Parserator, for this. This gist is, however, very tightly coupled with my codebase and I ought to turn it into a proper package. I even got typed JSON fields working!

• mysqueue - a job queue with Effect-TS and MySQL

  • Hacked together a job queue using Effect-TS and MySQL to use in Outroute. We were previously using BullMQ, but wanted something simpler that also didn’t require us to maintain a Redis instance. We had all our data on MySQL, hence we couldn’t use Postgres’s incredible pub/sub features.
  • This is what the API looks like now:

    import { z } from 'zod'
    import { Duration, Effect } from 'effect'
    import { createQueue, QueueContext, startWorkers } from "@texoport/mysqueue";
    import * as Schema from '@effect/schema'
    
    const something = Effect.gen(function* () {
      const ctx = yield* QueueContext;
      console.log(ctx.job)
    })
    
    
    const testQueue = createQueue(
      "test",
      Schema.Struct({
        name: Schema.String
      }),
      (payload) =>
        Effect.gen(function* () {
          const ctx = yield* QueueContext
          console.log("Do stuff")
          yield* something
          if (payload.name === "texoport") {
            return yield* ctx.retry(Duration.seconds(2))
          }
          console.log(payload.name, " complete");
        }),
    );
    
    await startWorkers([testQueue]);
    

  • Got retries, scheduling, and failure handling working
  • Next up: maybe add distributed locking if I ever get back to this
  • Once Effect-TS Cluster is released, I’d love to rewrite the whole thing from scratch to get all of the benefits of distributed workers.