New Jersey would create a formula for calculating a nonresident’s state tax liability on compensation from stock options under a proposed regulation issued Monday.