How are property rights established under Biblical law?
Quick answer: the same way that they are established under most modern law systems. You must document the boundaries of your plot of land by the use of some mutually acceptable marker, which can be witnessed, and by law cannot be moved.
The ancient landmarks
14 You shall not remove your neighbor’s landmark, which they of old time have set, in your inheritance which you shall inherit, in the land that YHWH your God gives you to possess. Deuteronomy 19:14WEB
Moving or removing such a landmark would be a form of theft.
How would a judge know if such a landmark had been moved? By witnesses: people who were prepared to testify and be cross-examined. Under Biblical law, you must have two or more witnesses in order to establish legal truth. It is literally the next verse after the one above:
15 One witness shall not rise up against a man for any iniquity, or for any sin that he sins. At the mouth of two witnesses, or at the mouth of three witnesses, shall a matter be established. Deuteronomy 19:15WEB
Jesus affirmed this:
31 “If I testify about myself, my witness is not valid. John 5:31WEB
17 It’s also written in your law that the testimony of two people is valid. John 8:17WEB
If you want to establish your ownership of a piece of property, then you need to mark it in a way that can be witnessed. Then you must get more than one person to witness that marker. If you are purchasing a piece of property from another person, then you must get that transaction/agreement witnessed by more than one person. Notice how Abraham bought the Cave of Machpelah:
9 that he may sell me the cave of Machpelah, which he has, which is in the end of his field. For the full price let him sell it to me among you as a possession for a burial place.” Genesis 23:9WEB ... 16 Abraham listened to Ephron. Abraham weighed to Ephron the silver which he had named in the hearing of the children of Heth, four hundred shekels of silver, according to the current merchants’ standard. 17 So the field of Ephron, which was in Machpelah, which was before Mamre, the field, the cave which was in it, and all the trees that were in the field, that were in all of its borders, were deeded 18 to Abraham for a possession in the presence of the children of Heth, before all who went in at the gate of his city. Genesis 23:16-18WEBGenesis 23:9,16-18
Transactions were often made at the city gates, so that anyone happening to be there might serve as witnesses.
Claim of prior ownership
If you believe that you own a particular piece of property, and someone else is currently "possessing" it, then you must make a legal claim of prior ownership, bring witnesses to those landmarks (or the previous position of removed landmarks), and allow a judge to decide your claim. Even if someone else is the currently documented owner of a particular piece of property, it is certainly possible for you to establish legally your prior ownership of that plot. But you must show actual evidence that you owned that plot, before you can remove the people who currently possess it.