>>What is the ultimate difference between a native speaker and a non-native speaker outside language mastery?

This is a bizarre question. What distinguishes native speakers is having acquired speaking ability in a given language naturally, from birth, rather than primarily through formal instruction, ie being native speakers. It's too late for you on that front, so language mastery is your only option. My advice is to study hard and master as much grammar and vocabulary as you can in a classroom setting--drills, etc., are very good tools for basic language acquisition--and then supplement this with several years living in Japan, using only Japanese. This will enable you to acquire natural speech patterns, idiomatic expressions, and other stuff that makes natives sound like natives and can't really be gained in a classroom.

The combination of in-school cramming and real-world use will get you closer than anything else to native-level mastery of a language. For example, after five years of formal in-school study of Thai and two years of living in a part of Thailand where not much English was spoken, I was often mistaken for a Thai person on the phone.

If you're looking to be mistaken for a Japanese person face-to-face, my advice is to combine all of the above steps, and to also have Japanese parents. Parents from another part of Northeast Asia would be a distant second choice.