Adventures with TinyGrad

Tinygrad source: https://github.com/tinygrad/tinygrad

Tinygrad docs: https://docs.tinygrad.org/

Written with help from good people in TinyGrad Discord, and a great deal of inspiration has been taken from mesozoic-egg.

Chapters

• 0 - Introduction
• 1 - UOps
• 2 - View and ShapeTracker
• 3 - The Pattern Matcher
• 4 - The .arange() insanity

Appendices

• A: helpers.py
• B: UOp Summary

Miscellaneous

• 1 - elf.py and the ELF format
• 2 - CUDA Runtime library

Reading order:

• Scroll through Appendix A after reading the intro. It will make reading the code much easier
• Scroll through Appendix B after reading the chapter of UOps

If you want to contribute, install nbdev

Run

nbdev_clean
nbdev_docs

I am not a madman for saying that it is likely that the code for Artificial General Intelligence is going to be tens of thousands of lines of code, not millions of lines of code.
This is code that conceivably one individual could write, unlike writing a new web browser or operating system.
And based on the progress that AI machine learning has made in the recent decade, it’s likely that the important things that we don’t know are relatively simple.

— John Carmack