Opinion: At least once, make a neural network without any packages
In the past year I’ve done two things that stand above all else in their value to my education. I’d recommend you do the same.
As the title suggests, I coded a neural network without using any external packages (except numpy). I think it’s alright to use numpy as it will only handle your matrix multiplication. Doing this allows you to really understand activation functions, loss functions, learning rates, and backwards propagation. The math (to me) was not that hard, some of it I did learn when I took calculus 3, but if you’ve finished calculus 1, you should be able to figure it out if you learn how to partial derivatives. The hard part for me was organizing all the matrices.
I also implemented a Markov chain Monte Carlo (during a bayesian statistics class) without using an external package. The math here is much harder to understand, though less difficult to code. I’d recommend being very well grounded in distribution theory and knowing what gibbs sampling is and how to do it before attempting this.