In Germany, the solar panels in commercial installations produce about 10% of their peak wattage averaged over a year. I guess these are installed in locations with good insolation. So at best you could assume some 2W on average from your 20W panel. But if it didn't even work in direct sunlight, the panel was probably broken.
In theory, couldn't you boost the input light by using a couple of mirrors (one capturing light below the panel, one above the solar-incident light shining down at a Summertime angle?) or some carefully position Fresnel-style lensing?
