They will ban books regardless of whether harmful books are banned.

Freedom of speech doesn’t extend to incitement of hatred. If it does, your laws don’t protect freedom of speech as much as they protect the freedom to call for, and eventually cause, genocide.