How are property rights established under Biblical law?
Quick answer: the same way that they are established under most modern law systems. You must document the boundaries of your plot of land by the use of some mutually acceptable marker, which can be witnessed, and by law cannot be moved.
The ancient landmarks
14 You shall not remove your neighbor’s landmark, which they of old time have set, in your inheritance which you shall inherit, in the land that YHWH your God gives you to possess. Deuteronomy 19:14WEB
Moving such a landmark would be a form of theft. How would a judge know if such a landmark had been moved? By witnesses: people who were prepared to testify and be cross-examined. Under Biblical law, you must have two or more witnesses in order to establish legal truth. It is literally the next verse after the one above:
15 One witness shall not rise up against a man for any iniquity, or for any sin that he sins. At the mouth of two witnesses, or at the mouth of three witnesses, shall a matter be established. Deuteronomy 19:15WEB
Jesus affirmed this:
31 “If I testify about myself, my witness is not valid. John 5:31WEB
and
17 It’s also written in your law that the testimony of two people is valid. John 8:17WEB
If you want to establish your ownership of a piece of property, then you need to mark it in a way that can be witnessed. Then you must get more than one person to witness that marker. If you are purchasing a piece of property from another person, then you must get that transaction/agreement witnessed by more than one person.
If you believe that you own a particular piece of property, and someone else is currently "possessing" it, then you must make a legal claim of prior ownership, brink witnesses to those landmarks (or the previous position of removed landmarks), and allow a judge to decide your claim. Even if someone else is the currently documented owner of a particular piece of property, it is certainly possible for you to establish legally your prior ownership of that plot. But you must show actual evidence that you owned that plot, before you can remove the people who currently possess it.