These worries aren't scaremongering, they're what I see directly on the ground as someone in a university, I'm afraid to say.
As to your first point, ERC funding grants are only one part of that picture, and I should also point out that what you've quoted applies specifically to the PI and requires the PI to be hosted by a European institution. That is to say, what it would allow for is a Professor or permanent academic fellow, the people already high up the academic food chain, to go to Europe and get a grant to do some work whilst
at a European university. So we couldn't actually get that money into UK universities (unless we applied to be an Associated Country, which is possible but costs money and we wouldn't get first dibs on funding pots any more so we'd almost certainly lose the fact that we're currently #1 in creating collaborative research projects in Europe). So what you've referenced describes actually quite a specific situation applying to the higher tier of academics, not a general principle that allows for all other sorts of extra-European collaboration.
The things I worry about are much wider than funding pools - higher travel costs coupled with the extreme funding restraint that's been applied to our sector will make conferences a lot more difficult particularly for PhD or Masters student researchers, who a) do a lot of the coalface work in many academic departments and b) are often surviving on very little money. It would also make it harder for us to shape the direction of European academic funding - even if we bought back into the ERC, we'd be paying in again and we'd lose the seat at the table for determining where research goes (Switzerland is in that position now for example), and compulsory free movement agreements are likely to be attached to any attempt to re-enter such programmes (again, this is currently a problem for Switzerland, who have ended up having their access to schemes severely restricted because of their refusal to implement such agreements; they're currently suspended from the Erasmus programme IIRC).
The community of academics I work and live around isn't big - it's a subsection of a couple of small departments in just one university. I still know talented academics from outside the EU who like it here and would like to work here, but are leaving the UK immediately at the end of their studies because our deportation and visa regime is now so harsh that they won't have time to apply for jobs here between finishing their studies and being forced to leave. We had a Birmingham researcher get
arrested by immigration police recently whilst going through the procedures for trying to stay in the UK. Also, if you think it's easy to hire staff from across the globe, it's absolutely not. The migration restrictions on getting researchers in from outside the EU are extremely onerous, I know scientists who've found it extremely difficult to get staff in from outside the EU, especially in non-permanent posts. It's a little easier if people are moving here for a permanent job, but that's just not how a lot of academia operates, we need freedom of movement to be able to function effectively.
I'm not necessarily saying that all that should sway your view, if you believe in national sovereignty at all costs then fine. But I'm afraid to say you're wholly wrong to call these concerns scaremongering, this is an issue right now for non-EU nationals, which after a Leave vote would have the EU nationals - of whom there are huge numbers working in our research sector - added to them. This won't on its own kill off the UK research sector, of course, and the financial issues could be solved with a very large cash injection into academia to counterbalance the losses, but I'm not holding my breath for that happening and it would certainly be less cost-effective than remaining in the system.