A Northern Ireland Office minister has indicated that opinion polls will determine if and when a border referendum is called.