Graham Potter said he "hasn't done enough at Chelsea to have too much good faith" as pressure mounts on him after the 2-0 defeat at Tottenham.