Without context it is just "go", and that would be an appropriate translation too. He meant "go (away)" and by no means "come with me".