Git is designed to facilitate sharing. Repos are a poor tool for managing secrets, especially intermixed with a general software project.

Use something else for secrets. Ideally you would generate the secret on the same server on which it will be used and not move it over the network (except for a one-off backup).