A normal frame rate is probably enough if you do it with groups of columns rather than a single column of pixels. That's what https://trains.jo-m.ch/ does with a Raspberry Pi camera, posted in https://news.ycombinator.com/item?id=35738987 with lots of questions and answers.
Oh wow that's neat! I honestly didn't think of using chunks instead of a single line and the result looks pretty good.
I love the fact the top (or first?) comment is by dllu, and looking on their webpage I saw the tram photo from Wikipedia! It's cool to see the photographer talking about their work. I think about that tram photo so much.
