In that case make sure you only remove the staged version, and add the file to your .gitignore to avoid making the same mistake a second time:
git reset filename # or git remove --cached filename
echo filename >> .gitingore # add it to .gitignore to avoid re-adding it
You can also remove files from the repository based on your .gitignore without deleting them from the local file system :
git rm --cached `git ls-files -i -X .gitignore`
Or, alternatively, on Windows Powershell:
git rm --cached $(git ls-files -i -X .gitignore)