If you are too close to the nest, they will send out scouts, and then you can expect an attack. Kill them first.