Among  the  challenges  in  the  recent  research  ofend-to-end  (E2E)  driving,  interpretability  and  distribution  shiftin the simulation-to-real (Sim2Real) have drawn considerableattention.  Because  of  low  interpretability,  we  cannot  clearlyexplain  the  causal  relationship  between  the  input  image  andthe control actions by the network. Moreover, the distributionshift problem in Sim2Real degrades the driving performanceof  the  policy  in  the  realworld  deployment.  In  this  paper,  wepropose  a  segmentation-based  classwise  disentangled  latentencoding  algorithm  to  cope  with  the  two  challenges.  In  theproposed  algorithm,  multi-class  segmentation  transfers  RGBimages in both simulation and real environments to the samedomain, while preserving the necessary information of objects ofprimary  classes,  such  as  pedestrian,  road,  and  cars,  for  drivingdecisions. Besides, in the class-wise disentangled latent encoding,segmented images are encoded to a latent vector, which improvesthe  interpretability  significantly,  since  the  state  input  has  astructured  format.  The  interpretability  improvement  is  testifiedby the t-stochastic neighbor embedding, image reconstructionand  the  causal  relationship  between  the  real  images  and  thecontrol  actions.  We  deploy  the  driving  policy  trained  in  thesimulation directly to an autonomous vehicle platform and show,to  the  best  of  our  knowledge,  the  first  demonstration  of  theRL-based E2E autonomous in various real environments