If you lived in the Mediterranean area like me, you really wouln't bother...
You can buy 800 year old olives from garden centers around 1500 € (transportation included). Of course you'll get a 2m diameter trunk tree...
But what this means is that there is a shotload of material around every corner, and since an olive takes around 15 years to mature before it grows to the sides, there is only so much time you can save by collecting them...