Many thanks to everyone for all the responses and I apologise for the late reply. After some experimenting, here are my thoughts on different aspects which may cause this phenomenon:
Cleaning: I daily backflush the machine with water and I did it also with some pullycaff several times along with some descaling during the past few weeks to make sure the everything is clean, hence I eliminated this factor.
Water: I use Brita cartridges for softening the water and I refill the tank before each use. I also tried mineral water (Highland Spring) and I also tasted the hot water itself coming out of the group and it tasted just fine so it is not the water IMO.
Taste buds: this is perhaps the most likely explanation, however, as I recall, whenever I am in Italy and I go to my favourite bars after lunch for a shot I always find the taste just the same delicious as in the morning. To justify my memory, today I went after lunch to a local Italian deli where I got my latest beans from and the same espresso from the same beans tasted just fine without this weird bitter taste or burnt smell. I went back home straight and my shot did taste badly although the extraction looked fine.
Extraction: I am a big fan of Italian espresso and Italian brew ratio - 14-15 g in, approx. 50 ml out in 25-30s splitted into two cups. As volume is a bit misleading due to the crema, I try to gain approx. 40 ml. The time for extracting this weight stays approximately the same for both the morning and the afternoon shot.
Beans: I know this goes against to what most of the members here like but I fell in love with espresso in Italy and the reason why I bought the Gaggia is that I want to get that taste back, my taste buds really cannot stand the light roasts I get in a third-wave coffee shop. I am not saying that is bad espresso, it simply comes down to preferences. I have a contact in a smaller roastery in Faenza, called Mokador, and I usually try and get their freshest roasts. As I mentioned before, I also got a bag from a local deli where I like the coffee and I am using that now until I get my next Mokador package. But the problem stands for both: in the morning I do get the chocolatey Italian taste that I desire with loads of crema (thanks to the 10-20 percent of Robusta) and in the afternoon I get this unpleasant, maybe bitter, rubbery taste.
Grinder: I use a Lelit Fred grinder, which is stepless, has stainless steel conical mills and does a pretty good job with keeping the grind consistent IMO. I don't think it overheats the beans as I use it once in the morning for 10-15s, so even if the burrs are a bit heated up there is no way they would not cool down in 3 hours. I always try to get rid of any remaining grinds but again - if those were the problem they would influence the morning shot even more.
Pressure: I did a mod on my new Classic and installed the pre-2015 adjustable OPV so I could set the pressure to 9bar (the 2019 OPV is not adjustable). I tried to do some low-pressure preinfusion and pressure profiling by overheating the boiler with the steam button before brewing and then opening-closing the steam valve during the extraction but that did not seem to take away the bad taste - and obviously induced a significant heat loss but I gave it a try anyway.
Temperature: this is something I could not fully investigate as I have not yet installed a PID - it may be that the boiler temperature fluctuates differently after the first use in the morning. I always turn it off in the morning and turn it on again after lunch. I tried 30min preheating and no flush and also long flush until brew light goes off, wait until it comes back then start extraction - I got different colours in the cup but the bad taste remained for both. I am going to install the PID though to make sure I create exactly the same environment in the morning as in the afternoon.
As you can see I tried to look at the problem from many perspectives. I skipped the morning shot several times- the afternoon one was still bad. Perhaps it really is my taste buds... anyway I cannot accept that simply I shouldn't enjoy my afternoon espresso so if anyone can recommend a further testing scheme or a suggestion where the problem lies I would be delighted to hear it!