﴿ قَالَ فَإِنَّا قَدْ فَتَنَّا قَوْمَكَ مِنۢ بَعْدِكَ وَأَضَلَّهُمُ ٱلسَّامِرِىُّ ﴾

“Said He: "Then [know that], verily, in thy absence We have put thy people to a test, and the Samaritan has led them astray."”

But those he supposed to be following him had remained behind for He exalted be He said ‘Indeed We tried your people after you that is after your departure from them and the Samaritan led them astray’ so they took to worshipping the golden calf.