for file in *.tar.gz; do tar xzvf "${file}" && rm "${file}"; done
Don't forget to quote your variables to account for funky filenames with whitespace.
Simply change the order of actions:
for i in *.tar.gz; do tar xzvf "$i" && rm -r "$i"done