You must have Tahlkora in your party to go to Garden of Seborhin if you have not finished the game yet.
If you have completed the Grand Court of Sebelkeh, then the entrance to the Garden of Seborhin instead takes you to Nightfallen Garden. You must reach this boss before completing that mission, or beat the game to revert it back to normal.
You can also join someone who has Garden of Seborhin instead of Nightfallen Garden. This will result in being able to go back into the Garden of Seborhin.