Local Wallet runs on port 8125, pool runs on port 8124.
I figured it out with help of @FrilledShark, needed to use port 8125 and use JMiner not Blago's miner as couldn't find where to place my passphrase in Blago's and its config file keeps restoring whenever you fire it from wallet (need to start it from command prompt not wallet).