Chelsea boss Graham Potter acknowledged fans' frustrations after a home loss to Southampton but said critics were wrong to put all the blame on him.