Can someone help I’m not sure if I got this problem right

When a transformation maps a figure to itself, each point on the figure must map to itself, and not just the out line that "appears" to map to itself.

To see which one is the correct answer, we only have to simplify the composite transformation to a simple one, and then apply the simple transformation to the four corners of the figure.
Recall that composite transformations start from right to left, we will name the points ABCD clockwise from the top-left corner.

Also, note the following identities:
[tex]R_{180,a,b}(x,y) [/tex]
[tex]=(x+a,y+b) \circle (-x,-y) \circle (x-a,y-b)[/tex]
=(2a-x, 2b-y)
For example, rotate point C(7,-9) by 180 about (5,5) 
[tex]R{180,5,5}(7,-9) = (2*5-7, 2*5-(-9)=(3,19)[/tex]

Similarly,
[tex]Sy_{y=b}(x,y) [/tex]
[tex]=(x,y+b)\circle(x,-y)\circle(x,y-b)[/tex]
[tex]=(x,2b-y)[/tex]

The four points of the rectangle are
A(5,-5)
B(7,-5)
C(7,-9)
D(5,-9)

Applying the above transformations, we will find

Case A Rotate 180 about (5,5), reflect across y=5
we can use the shorthand (x,y)->(10-x,y) to draw the final position
Since x does not map to x, and y does not map to y, this is not the correct transformation. The image of points A, B, C, D are
A(5,-5)
B(3,-5)
C(3,-9)
D(5,-9)
You can locate the image either by the above results, or actually transform (first rotate, then reflect) and still get the same results. 

Case B: rotate 180 about (6,-7), reflect across x=5 gives the transformation
(x,y)->(12-x, y+24) giving the image
A(7,19)
B(5,19)
C(5,15)
D(7,15)

Case C: rotate 180 about (5,5), reflect across y=-7 gives the transformation
(x,y)->(10-x,y-24) and the image
A(5,-29)
B(3,-29)
C(3,-33)
D(5,-33)

Case D: rotate 180 about (6,-7), reflect across y=-7 gives the transformation
(x,y)->(12-x,y) and the image
A(7,-5)
B(5,-5)
C(5,-9)
D(7,-9)

We see that in NONE of the cases that we have all points mapping onto itself, i.e. (x,y)->(x,y).  So we can say that NONE of the answer choices will map all points onto itself.

HOWEVER, the closest one is case D, where the image actually coincides with the preimage, giving the impression that it maps onto itself.
An examination of the coordinates will reveal that pairs of nodes actually coincide, A and B", B and A", C and D", D and C", giving the APPEARANCE that the rectangle maps onto itself.

Attached is the process of making the transformation graphically, for case A.

Note: A single reflection will never map ALL points onto itself  (except those that lie on the line of reflection).  So we could have foreseen the abnormal behaviour.  On the other hand, combinations of translations, rotations, or even number of reflections COULD end up mapping all points on the preimage onto itself.