Have you raised a set of false expectations, partly through an attempt to keep an option open? It's now becoming clear that you don't want what you once thought you might want. What you don't want to do, though, is say as much and make someone upset or angry. But the longer you leave it the trickier things will become. You can't just hope that fate will step in and fix things so that you don't have to take responsibility for a choice you have made. It's time to be strong and clear, then all will be fine.