A popular brunch spot in Downtown Madison will soon be closing its doors permanently, the owners announced on Facebook Sunday.