The reasoning for the custom of asking for a tzadik's intercession was explained to me growing up, that because the neshamah of a tzadik continues to be present at their burial site it isn't avodah zara to ask for their intercession.
But there are several keavrim from biblical times where there's a question as to whether or not the said tzadikim are actually buried where it is popularly believed they are. For example, there are five (and possibly more) kevarim that are believed by local Jews in the following regions to be the tomb of Daniel. Two in the Khuzestan region of Iran, one in the city of Kirkuk in Iraq, one in Samarkand in Uzbekistan, and one in Jorf El-Yhoudi in Morocco. Or for another example, many historians find it doubtful as to whether or not Esther and Mordechai are truly buried at their purported location in Hamadan given that the edifice is believed to have been built in the twelfth century. In cases like these where one can be doubtful if not highly suspicious of the historicity of these locations, is there any point to visit and daven at these sights anyway?