I found out that it is truly impossible to have multiple docker files in one folder, but there is a workaround. I have created a separate subfolders with each having its own dockerfile for either windows or linux in it and nothing else. In the YAML file for the pipeline that creates those docker images, I have added a step before docker compose, that moves the dockerfile out of the subfolder and into the main folder where the relevant code files are. It is possible to have only one YAML file and create an IF statement that will select the correct dockerfile based on parameters of the pipeline run.