What's The Problem With Covid Passports?

When the idea of Coronovirus Passports was first mentioned i thought it was quite a good idea, and not just for International Travel but for places which attract large crowds such as football matches and music concerts.
Some people didn't agree, stating it would create a two tier system of haves and have nots but if i was sat in a metal container with 200 other people on a six hour flight, then i would be happier knowing none of them are breathing out Covid-19 for the rest of us to breath in.
The Government took a break today from trying to justify why they are delaying, yet again, their hotel quarantine policy to say that they don't have plans to bring in the passport scheme and it is up for other nations to decide what they do.
Greece, Denmark and Sweden have announced that they are introducing such a measure and will require such a certificate to state that the visitor has been immunised before they will let anybody in which may cause a problem if Brit's want to go there but have no record to show they have had both the required jabs.
I would also be happy to know that we are not allowing people in who could potentially start the whole thing up again once we have gone through a year of lock-downs and 120,000 deaths to eradicate it so what we need is an agreement all around the World but it will probably fall to the airways to introduce the passports before people can fly although that doesn't stop people traveling by other means.   
The voices against it include a petition which states that: 'the immunity passports could be used to restrict the rights of people who have refused a Covid-19 vaccine, which would be unacceptable' but to me, restricting people who have not been vaccinated, is the whole point.
Most of us have the NHS Covid 19 app which could show the dates of vaccinations, we don't need to hand over our medical records to the Government, just something unique to each person to say yep, their safe and i have no problem with that whatsoever and am surprised that some people do.