Roger Lam

Change is constant, even for Fields Medalists

Howdy, hope all is well!

Wanted to first note some professional milestones that happened in the past few months and then get into AI current events.

Work Wins

We did it! After a year and a half of putting our heads down and putting in the work, we launched MN Paid Leave "ahead of schedule and under budget."

"...The implementation came in $70 million below its budget, savings that will be redirected into the program’s trust fund."

That's what I'm talking about!!! What an incredible result by everyone involved!

CMD+C, CMD+V?

Which made it even more bittersweet to switch teams but the moment fit the opportunity.

Nava is investing heavily in Strata, our free, open-source software stack built to replicate past success.

Specifically, states are facing legislative changes to reduce error rate and collect community engagement reporting.

After joining the Strata team a month or so ago, I can feel how dedicated everyone is to helping states meet this moment.

If this interests you, we have a PM role open on the OSCER community engagement reporting side and a range of other public benefit oriented roles.

PyTorch Certificate

I've been focusing my free time on becoming more fluent in PyTorch and highly recommend the DeepLearning.ai PyTorch for Deep Learning course. As part of my New Years resolutions, I've been making a habit of doing something PyTorch related every day. Following a course helped with the mental load of deciding what to do and each step in the course is contained to ~10 min videos and guided hands-on notebooks. The three courses took me about 3 months. Give it a try!

AI Signal from the Noise

Dropping a few conversations that I enjoyed recently.

Terence Tao, new research, and Foundation

Terence Tao on Dwarkesh had a quote that "productivity is not quite a one dimensional quantity" - love it. Terence also notes that we're in a time of change - and how he would prefer a much more boring, quiet era as thing were 10, 20 years ago. Still you have to embrace that there's going to be a lot of change. Also, with the change comes a lot of non-traditional opportunities to learn.

That's one of my favorite parts of software and software engineering. Forget about gatekeeping. Make things. Get good. And find places and opportunites that value creative grit and willpower.

I look forward to the future mini-neo labs that make the power of LLMs accessible to all.

The Foundation for Science and AI Research (SAIR) was launched Tao and Damek Davis. Think ARC-AGI or Kaggle but for fundamental math and science. The first competition is to create a prompt and math cheatsheet for LLMs to solve "equational implication over magmas" - aka Does Equation 1 imply Equation 2? Sounds fun to take a crack at.

Some cool architectural research

Neural Thickets suggest there are dense pockets of experts around your pretrained weight location and they're pretty easy to find.

What this might mean: Terence mentioned labs liking astronomers because they look for significance with little data. This feels smiliar - looking for experts or planets. This could be key to making small language models punch above their weight class.

Deepseek showing again it's the real Open AI with Engrams. Adding a module to the transformer to detect multi-token patterns like "piece of cake" or "Alexander the Great" improves performance. There more to it to make it efficient and they not afraid to share.

Attention Residuals adds attention to residual connections so layers can be more flexible in leveraging earlier representations rather than only looking at what it receives from the previous layer. This video explains it way better.

Lots to reproduce and try!

Personalized Learning

Maybe Claude has been good at this for a while but it was really good at developing a learning plan around LLM research.

You can tell it "Hey I have an hour each day, two hours on weekends and I want to learn X. Please create a plan." and it'll do a pretty good job. I'm keeping track of personal learning plans in this repo.

Conclusion

Lots to dig into as always. Small steps count. Keep at it!