Yes, you can use it at many stores regardless if it says Visa or Mastercard. That said, there are good reasons to just get cash from the ATM because that limits your spending. If you only have $20.00 in your pocket, that is all you can spend. If you use the debit feature, you are likely to spend much more. That is why the merchants accept the cards.
The thing is that you MUST write each purchase in your check register as each time you use it at a store, that is an INSTANT CHECK. Meaning you better have had the money in the checking account yesterday.
The machines that you see at the store have an option DEBIT or CREDIT meaning debit card (ATM card) or CREDIT (meaning credit card.)
The ATM card without a Visa or Mastercard logo does have limitations. The merchant must be a member of a network that knows what to do with the card information that is transmitted when you use the card. Some merchants may not be. You have to enter your PIN in the machine at the time of purchase. .. for every purchase. Therefore you have to take precautions to keep other people from seeing what your PIN is.
Talk to your banker. I know the ATM debit VISA or debit Mastercard has protections if someone else uses your card, but I do NOT know what protection you have against fraud when you use the ATM card as a debit card.
I've taken a brief look at your bank's website. It appears that IF you decide to use your card, you should consider a checking Rewards card which has a Visa Logo. Again, talk to your banker. Do not decide this on the basis of the answers here.
IN EITHER CASE, REMEMBER YOUR USE OF THE CARD IS AN INSTANT CHECK. YOU NEED TO WRITE IT DOWN IN YOUR CHECK REGISTER.