Richard Bennett called the ban a retrograde step that should be reversed but vowed he would continue to document human rights violations in Afghanistan.
"The Taliban's public announcement that they will no longer grant me access to Afghanistan is a step backwards and sends a concerning signal about their engagement with the United Nations and the international community on human rights," he said in a statement.
Bennett marked two years in the role on May 1.
Since returning to power in August 2021, Taliban authorities have enforced a rule based on a strict interpretation of Islamic law.
Women have borne the brunt of restrictions the UN has labelled "gender apartheid", which have pushed them from public life.
Taliban authorities, who have not been recognised by any country, have dismissed criticism of their policies from the UN and the international community.
"Despite this announcement, the substance of which was conveyed to me previously, I will continue to engage with the people of Afghanistan, both inside and outside the country, as well as other relevant stakeholders, noting that I have not travelled to Afghanistan for over a year," said Bennett.
"I will also continue to document human rights violations and abuses and advocate for improvements.
"I remain committed to the people of Afghanistan and to supporting a stable, inclusive and prosperous country at peace with itself and its neighbours."
UN special rapporteurs are independent experts appointed by the UN Human Rights Council in Geneva. They therefore do not speak for the United Nations.
"I have consistently sought to engage transparently with the de facto authorities, offering a critical assessment of the human rights situation in Afghanistan, as is my mandated responsibility, and making concrete, practical recommendations for improvement and offering technical assistance," said Bennett.
"Where there are differences of view, constructive dialogue is the answer.
"I urge the Taliban to reverse their decision and reiterate my willingness and availability to travel to Afghanistan."