It's actually kind of insane. With a legacy pat you can curl an image as a user but with a finegrained pat even with all the permissions you can't.

So anyway I didn't solve that problem. It just works for public repos. Private repos can upload their images on the agent page.