Pointer Graph Networks